Jul 30 10:48:14.159421 ip-10-0-128-243 systemd[1]: Starting Kubernetes Kubelet...
Jul 30 10:48:14.577050 ip-10-0-128-243 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 10:48:14.577050 ip-10-0-128-243 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 30 10:48:14.577050 ip-10-0-128-243 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 10:48:14.577050 ip-10-0-128-243 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 30 10:48:14.577050 ip-10-0-128-243 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 10:48:14.577792 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.577689    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 30 10:48:14.581279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581264    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 10:48:14.581279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581279    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581283    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581288    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581291    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581295    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581297    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581301    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581303    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581306    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581310    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581314    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581317    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581320    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581323    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581326    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581329    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581332    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581334    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581340    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581343    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 10:48:14.581338 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581346    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581349    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581351    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581354    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581357    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581360    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581363    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581367    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581370    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581373    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581377    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581381    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581384    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581387    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581389    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581392    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581394    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581397    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581399    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 10:48:14.581812 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581402    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581404    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581407    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581409    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581411    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581414    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581416    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581419    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581421    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581424    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581426    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581429    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581431    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581434    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581437    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581440    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581442    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581445    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581447    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 10:48:14.582296 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581450    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581453    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581456    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581458    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581461    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581463    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581466    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581468    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581471    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581473    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581477    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581479    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581482    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581484    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581487    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581489    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581491    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581494    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581497    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581499    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 10:48:14.583078 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581501    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581504    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581506    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581508    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581511    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581513    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581516    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581519    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581522    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581524    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581527    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581529    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581532    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581535    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581537    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581540    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581542    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581544    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581547    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 10:48:14.583658 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.581550    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 10:48:14.585062 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585022    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 10:48:14.585062 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585061    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 10:48:14.585062 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585065    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585068    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585072    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585078    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585082    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585086    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585090    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585093    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585110    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585115    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585120    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585123    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585126    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585128    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585131    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585134    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585136    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585139    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585141    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585144    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 10:48:14.585193 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585147    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585150    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585152    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585155    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585159    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585162    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585165    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585168    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585170    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585173    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585175    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585178    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585181    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585183    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585188    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585191    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585194    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585197    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585201    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 10:48:14.585703 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585203    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585206    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585209    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585212    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585214    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585217    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585220    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585222    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585225    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585229    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585231    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585234    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585237    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585240    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585242    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585245    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585247    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585250    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585252    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585257    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 10:48:14.586198 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585259    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585263    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585267    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585270    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585272    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585275    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585278    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585280    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585283    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585285    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585288    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585290    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585293    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585296    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585298    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585301    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585303    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585306    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585308    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585311    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 10:48:14.586691 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585313    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585316    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585320    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585323    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585326    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585328    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585331    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585334    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585336    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585339    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585342    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585345    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585348    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585350    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585353    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585356    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585358    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585360    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585740    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 10:48:14.587211 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585746    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585749    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585752    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585755    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585757    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585760    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585762    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585765    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585767    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585770    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585773    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585775    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585778    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585780    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585782    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585785    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585788    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585790    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585793    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585795    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 10:48:14.587698 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585798    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585800    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585803    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585806    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585809    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585812    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585815    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585817    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585819    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585822    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585825    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585827    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585830    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585832    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585835    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585837    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585840    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585842    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585845    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585848    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 10:48:14.588207 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585850    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585853    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585855    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585858    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585860    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585863    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585865    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585867    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585872    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585874    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585877    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585879    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585882    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585884    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585887    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585890    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585893    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585895    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585898    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585900    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 10:48:14.588701 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585903    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585906    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585909    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585911    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585913    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585916    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585919    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585921    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585924    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585926    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585929    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585931    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585933    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585936    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585940    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585943    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585946    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585949    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585952    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 10:48:14.589274 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585955    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585959    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585963    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585966    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585968    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585971    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585973    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585976    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585978    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585981    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585984    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585986    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585989    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585991    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585994    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585996    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.585999    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586001    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586004    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586076    2571 flags.go:64] FLAG: --address="0.0.0.0"
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586088    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 30 10:48:14.589727 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586094    2571 flags.go:64] FLAG: --anonymous-auth="true"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586114    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586118    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586122    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586126    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586129    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586132    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586135    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586138    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586142    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586145    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586148    2571 flags.go:64] FLAG: --cgroup-root=""
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586151    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586153    2571 flags.go:64] FLAG: --client-ca-file=""
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586157    2571 flags.go:64] FLAG: --cloud-provider="external"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586160    2571 flags.go:64] FLAG: --cluster-dns="[]"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586165    2571 flags.go:64] FLAG: --cluster-domain=""
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586167    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586170    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586174    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586177    2571 flags.go:64] FLAG: --container-log-max-files="5"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586181    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586184    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 30 10:48:14.590239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586187    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586189    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586192    2571 flags.go:64] FLAG: --contention-profiling="false"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586195    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586201    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586204    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586208    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586212    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586215    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586218    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586220    2571 flags.go:64] FLAG: --enable-load-reader="false"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586223    2571 flags.go:64] FLAG: --enable-server="true"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586226    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586230    2571 flags.go:64] FLAG: --event-burst="100"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586233    2571 flags.go:64] FLAG: --event-qps="50"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586236    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586239    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586242    2571 flags.go:64] FLAG: --eviction-hard=""
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586245    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586248    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586251    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586254    2571 flags.go:64] FLAG: --eviction-soft=""
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586256    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586259    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586263    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 30 10:48:14.590826 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586266    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586268    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586271    2571 flags.go:64] FLAG: --fail-swap-on="true"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586274    2571 flags.go:64] FLAG: --feature-gates=""
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586278    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586281    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586284    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586287    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586290    2571 flags.go:64] FLAG: --healthz-port="10248"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586293    2571 flags.go:64] FLAG: --help="false"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586296    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-128-243.ec2.internal"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586301    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586304    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586306    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586310    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586313    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586316    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586319    2571 flags.go:64] FLAG: --image-service-endpoint=""
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586322    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586325    2571 flags.go:64] FLAG: --kube-api-burst="100"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586328    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586331    2571 flags.go:64] FLAG: --kube-api-qps="50"
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586334    2571 flags.go:64] FLAG: --kube-reserved=""
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586336    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 30 10:48:14.591495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586339    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586342    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586345    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586348    2571 flags.go:64] FLAG: --lock-file=""
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586350    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586353    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586357    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586362    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586366    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586369    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586372    2571 flags.go:64] FLAG: --logging-format="text"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586375    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586380    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586383    2571 flags.go:64] FLAG: --manifest-url=""
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586386    2571 flags.go:64] FLAG: --manifest-url-header=""
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586390    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586393    2571 flags.go:64] FLAG: --max-open-files="1000000"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586397    2571 flags.go:64] FLAG: --max-pods="110"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586400    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586405    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586407    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586410    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586413    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586416    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586419    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 30 10:48:14.592089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586426    2571 flags.go:64] FLAG: --node-status-max-images="50"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586429    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586432    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586435    2571 flags.go:64] FLAG: --pod-cidr=""
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586438    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586443    2571 flags.go:64] FLAG: --pod-manifest-path=""
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586446    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586449    2571 flags.go:64] FLAG: --pods-per-core="0"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586452    2571 flags.go:64] FLAG: --port="10250"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586455    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586457    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0464cc0c621b77dbc"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586461    2571 flags.go:64] FLAG: --qos-reserved=""
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586464    2571 flags.go:64] FLAG: --read-only-port="10255"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586467    2571 flags.go:64] FLAG: --register-node="true"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586469    2571 flags.go:64] FLAG: --register-with-taints=""
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586473    2571 flags.go:64] FLAG: --registry-burst="10"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586476    2571 flags.go:64] FLAG: --registry-qps="5"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586479    2571 flags.go:64] FLAG: --reserved-cpus=""
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586482    2571 flags.go:64] FLAG: --reserved-memory=""
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586485    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586490    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586493    2571 flags.go:64] FLAG: --rotate-certificates="false"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586496    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586498    2571 flags.go:64] FLAG: --runonce="false"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586501    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 30 10:48:14.592719 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586504    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586508    2571 flags.go:64] FLAG: --seccomp-default="false"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586511    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586514    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586517    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586520    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586523    2571 flags.go:64] FLAG: --storage-driver-password="root"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586526    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586529    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586531    2571 flags.go:64] FLAG: --storage-driver-user="root"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586534    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586537    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586540    2571 flags.go:64] FLAG: --system-cgroups=""
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586543    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586548    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586551    2571 flags.go:64] FLAG: --tls-cert-file=""
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586554    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586561    2571 flags.go:64] FLAG: --tls-min-version=""
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586563    2571 flags.go:64] FLAG: --tls-private-key-file=""
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586566    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586569    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586572    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586575    2571 flags.go:64] FLAG: --v="2"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586578    2571 flags.go:64] FLAG: --version="false"
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586583    2571 flags.go:64] FLAG: --vmodule=""
Jul 30 10:48:14.593356 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586586    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.586589    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586701    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586705    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586708    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586711    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586714    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586716    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586720    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586723    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586726    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586728    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586731    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586734    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586737    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586739    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586742    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586744    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586747    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 10:48:14.594001 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586749    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586752    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586755    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586757    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586760    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586762    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586765    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586768    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586770    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586773    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586776    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586778    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586781    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586784    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586787    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586791    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586793    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586797    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586800    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 10:48:14.594478 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586803    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586806    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586810    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586813    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586816    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586818    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586821    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586824    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586826    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586829    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586832    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586834    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586837    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586841    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586845    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586848    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586851    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586854    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586857    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 10:48:14.594941 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586861    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586863    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586866    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586869    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586872    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586875    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586878    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586880    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586884    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586888    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586891    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586894    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586896    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586899    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586902    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586906    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586908    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586911    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586913    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586916    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 10:48:14.595470 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586918    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586921    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586924    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586926    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586929    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586931    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586934    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586936    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586939    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586941    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586944    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586947    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586950    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586952    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586955    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586957    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586960    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586962    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586965    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586968    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 10:48:14.595962 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586970    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586974    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586977    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.586980    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.587653    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.594196    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.594214    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.594227    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.594233    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594286    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594291    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594294    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594297    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594300    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 10:48:14.596453 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594302    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594305    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594308    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594311    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594314    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594316    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594319    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594322    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594325    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594327    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594330    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594332    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594335    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594338    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594340    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594343    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594345    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594348    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594351    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594353    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 10:48:14.596802 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594356    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594358    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594362    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594366    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594370    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594373    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594376    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594378    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594381    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594385    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594388    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594392    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594394    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594397    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594399    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594402    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594405    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594407    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594410    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 10:48:14.597323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594428    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594432    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594435    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594438    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594441    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594444    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594448    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594451    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594454    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594457    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594460    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594463    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594465    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594468    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594471    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594474    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594477    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594481    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594490    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594493    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 10:48:14.597801 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594497    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594500    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594503    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594506    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594509    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594518    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594522    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594525    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594528    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594531    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594533    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594536    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594539    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594541    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594544    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594547    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594549    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594552    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594554    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594557    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 10:48:14.598372 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594559    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594561    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594564    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594567    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594569    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594572    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594574    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594577    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594579    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594583    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594586    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594589    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594592    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594594    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594597    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 10:48:14.599279 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.594602    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594709    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594713    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594716    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594719    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594722    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594724    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594727    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594729    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594732    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594735    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594737    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594740    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594743    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594745    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594748    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594750    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594752    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594755    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 10:48:14.599680 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594757    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594760    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594762    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594765    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594767    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594770    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594772    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594775    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594778    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594781    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594784    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594787    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594790    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594792    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594795    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594797    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594800    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594803    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594805    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594808    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 10:48:14.600270 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594811    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594813    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594816    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594818    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594821    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594823    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594826    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594829    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594831    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594834    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594836    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594840    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594843    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594846    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594849    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594852    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594854    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594857    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594860    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594863    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 10:48:14.600779 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594865    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594868    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594870    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594873    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594876    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594878    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594881    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594883    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594886    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594888    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594892    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594896    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594899    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594902    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594905    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594907    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594910    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594913    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594916    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 10:48:14.601477 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594919    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594922    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594924    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594927    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594929    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594932    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594934    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594937    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594940    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594942    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594945    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594947    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594950    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594953    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594955    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594958    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594960    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594963    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594966    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594968    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 10:48:14.602004 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594971    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 10:48:14.602549 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:14.594973    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 10:48:14.602549 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.594978    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 10:48:14.602549 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.595068    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 30 10:48:14.602549 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.596945    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 30 10:48:14.602549 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.598141    2571 server.go:1019] "Starting client certificate rotation"
Jul 30 10:48:14.602549 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.598252    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 10:48:14.602549 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.598303    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 10:48:14.602549 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.600410    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 10:48:14.602549 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.601489    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 10:48:14.607874 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.607860    2571 log.go:25] "Validated CRI v1 runtime API"
Jul 30 10:48:14.610245 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.610227    2571 log.go:25] "Validated CRI v1 image API"
Jul 30 10:48:14.610993 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.610978    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 30 10:48:14.613654 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.613634    2571 fs.go:135] Filesystem UUIDs: map[4276c6f7-f948-430d-9d0a-cd4d382dbeb0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e05e7cc3-0995-4dd7-a7df-53e91e335970:/dev/nvme0n1p4]
Jul 30 10:48:14.613708 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.613656    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 30 10:48:14.619219 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.619115    2571 manager.go:216] Machine: {Timestamp:2026-07-30 10:48:14.617761964 +0000 UTC m=+0.338646217 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099931 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2426cdc7867e9a802a433109b382f9 SystemUUID:ec2426cd-c786-7e9a-802a-433109b382f9 BootID:6b45ad68-786e-4e9c-83b2-7b93dbfa3258 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:98:8a:00:f2:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:98:8a:00:f2:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:68:4c:27:c0:d1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 30 10:48:14.619219 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.619211    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 30 10:48:14.619340 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.619279    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 30 10:48:14.622171 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.622141    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 30 10:48:14.622310 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.622175    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-243.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 30 10:48:14.623311 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.623300    2571 topology_manager.go:138] "Creating topology manager with none policy"
Jul 30 10:48:14.623343 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.623314    2571 container_manager_linux.go:309] "Creating device plugin manager"
Jul 30 10:48:14.623343 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.623322    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 10:48:14.623944 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.623934    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 10:48:14.625187 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.625177    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 30 10:48:14.628289 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.628277    2571 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 10:48:14.628469 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.628459    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 30 10:48:14.629063 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.629055    2571 kubelet.go:486] "Attempting to sync node with API server"
Jul 30 10:48:14.629108 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.629068    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 30 10:48:14.629108 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.629082    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 30 10:48:14.629108 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.629107    2571 kubelet.go:398] "Adding apiserver pod source"
Jul 30 10:48:14.629213 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.629116    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 30 10:48:14.629477 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.629464    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 10:48:14.629521 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.629487    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 10:48:14.632554 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.632538    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 30 10:48:14.632916 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.632898    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 10:48:14.634447 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.634431    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 30 10:48:14.634521 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.634470    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 30 10:48:14.635930 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.635915    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 30 10:48:14.636004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.635936    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 30 10:48:14.636004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.635945    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 30 10:48:14.636004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.635953    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 30 10:48:14.636004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.635966    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 30 10:48:14.636004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.635975    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 30 10:48:14.636004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.635988    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 30 10:48:14.636004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.635997    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 30 10:48:14.636004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.636007    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 30 10:48:14.636276 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.636016    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 30 10:48:14.636276 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.636029    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 30 10:48:14.636276 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.636038    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 30 10:48:14.636798 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.636787    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 30 10:48:14.636850 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.636801    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 30 10:48:14.639715 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.639699    2571 server.go:1268] "Started kubelet"
Jul 30 10:48:14.639797 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.639768    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 30 10:48:14.640058 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.639959    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 30 10:48:14.640058 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.640053    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 30 10:48:14.640515 ip-10-0-128-243 systemd[1]: Started Kubernetes Kubelet.
Jul 30 10:48:14.640743 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.640534    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-243.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 30 10:48:14.640743 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.640587    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-243.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 30 10:48:14.644297 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.644271    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 30 10:48:14.644834 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.644796    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 30 10:48:14.645234 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.645199    2571 server.go:310] "Adding debug handlers to kubelet server"
Jul 30 10:48:14.645412 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.645370    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 30 10:48:14.649850 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.649006    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-243.ec2.internal.18c70b90fdf66e18  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-243.ec2.internal,UID:ip-10-0-128-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-243.ec2.internal,},FirstTimestamp:2026-07-30 10:48:14.639672856 +0000 UTC m=+0.360557108,LastTimestamp:2026-07-30 10:48:14.639672856 +0000 UTC m=+0.360557108,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-243.ec2.internal,}"
Jul 30 10:48:14.650220 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.650200    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 30 10:48:14.650303 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.650214    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 30 10:48:14.650678 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.650595    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:14.651123 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651094    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 30 10:48:14.651123 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651125    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 30 10:48:14.651268 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651232    2571 reconstruct.go:97] "Volume reconstruction finished"
Jul 30 10:48:14.651268 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651238    2571 reconciler.go:29] "Reconciler: start to sync state"
Jul 30 10:48:14.651268 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651261    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 30 10:48:14.651406 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651319    2571 factory.go:55] Registering systemd factory
Jul 30 10:48:14.651406 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651365    2571 factory.go:223] Registration of the systemd container factory successfully
Jul 30 10:48:14.651605 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651584    2571 factory.go:153] Registering CRI-O factory
Jul 30 10:48:14.651605 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651605    2571 factory.go:223] Registration of the crio container factory successfully
Jul 30 10:48:14.651716 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651660    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 30 10:48:14.651716 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651683    2571 factory.go:103] Registering Raw factory
Jul 30 10:48:14.651716 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.651703    2571 manager.go:1196] Started watching for new ooms in manager
Jul 30 10:48:14.652340 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.652323    2571 manager.go:349] Starting recovery of all containers
Jul 30 10:48:14.662357 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.662339    2571 manager.go:354] Recovery completed
Jul 30 10:48:14.662448 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.662353    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-243.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 30 10:48:14.662624 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.662601    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 30 10:48:14.666261 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.666249    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 10:48:14.667950 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.667930    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 10:48:14.667995 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.667963    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 10:48:14.667995 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.667976    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientPID"
Jul 30 10:48:14.668300 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.668274    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f94wp"
Jul 30 10:48:14.668425 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.668408    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 30 10:48:14.668425 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.668420    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 10:48:14.668536 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.668424    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 30 10:48:14.668536 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.668495    2571 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 10:48:14.669070 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.669053    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 10:48:14.669157 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.669084    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 10:48:14.669157 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.669112    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientPID"
Jul 30 10:48:14.670632 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.670619    2571 policy_none.go:49] "None policy: Start"
Jul 30 10:48:14.670672 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.670636    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 30 10:48:14.670672 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.670647    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 30 10:48:14.671564 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.671554    2571 policy_none.go:47] "Start"
Jul 30 10:48:14.679876 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.679722    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-243.ec2.internal.18c70b90ffa5e739  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-243.ec2.internal,UID:ip-10-0-128-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-243.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-243.ec2.internal,},FirstTimestamp:2026-07-30 10:48:14.667949881 +0000 UTC m=+0.388834133,LastTimestamp:2026-07-30 10:48:14.667949881 +0000 UTC m=+0.388834133,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-243.ec2.internal,}"
Jul 30 10:48:14.680935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.680914    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f94wp"
Jul 30 10:48:14.706631 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.706614    2571 manager.go:337] "Starting Device Plugin manager"
Jul 30 10:48:14.706708 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.706641    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 30 10:48:14.706708 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.706650    2571 server.go:87] "Starting device plugin registration server"
Jul 30 10:48:14.706849 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.706837    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 30 10:48:14.706906 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.706853    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 30 10:48:14.706964 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.706947    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 30 10:48:14.707044 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.707032    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 30 10:48:14.707111 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.707046    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 30 10:48:14.707504 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.707488    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 30 10:48:14.707586 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.707527    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:14.807280 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.807260    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 10:48:14.808836 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.808820    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 10:48:14.808923 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.808852    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 10:48:14.808923 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.808866    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientPID"
Jul 30 10:48:14.808923 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.808897    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-243.ec2.internal"
Jul 30 10:48:14.819779 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.819760    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-243.ec2.internal"
Jul 30 10:48:14.819848 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.819781    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-243.ec2.internal\": node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:14.847823 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.847779    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 30 10:48:14.848873 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.848857    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 30 10:48:14.848946 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.848890    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 30 10:48:14.848946 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.848909    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 30 10:48:14.848946 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.848940    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 30 10:48:14.850161 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.850145    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:14.851155 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.851133    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 10:48:14.949598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.949567    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal"]
Jul 30 10:48:14.949693 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.949637    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 10:48:14.950653 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.950638    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:14.951058 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.951044    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 10:48:14.951140 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.951068    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 10:48:14.951140 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.951078    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientPID"
Jul 30 10:48:14.952233 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.952222    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 10:48:14.952282 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.952240    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db8ff15800a49eb6cf75db11d8895b81-config\") pod \"kube-apiserver-proxy-ip-10-0-128-243.ec2.internal\" (UID: \"db8ff15800a49eb6cf75db11d8895b81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:14.952352 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.952339    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:14.952385 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.952369    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 10:48:14.952840 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.952827    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 10:48:14.952840 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.952835    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 10:48:14.952923 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.952852    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 10:48:14.952923 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.952857    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 10:48:14.952923 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.952861    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientPID"
Jul 30 10:48:14.952923 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.952869    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientPID"
Jul 30 10:48:14.953839 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.953827    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:14.953898 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.953848    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 10:48:14.954458 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.954442    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 10:48:14.954526 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.954471    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 10:48:14.954526 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:14.954485    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeHasSufficientPID"
Jul 30 10:48:14.980610 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.980590    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-243.ec2.internal\" not found" node="ip-10-0-128-243.ec2.internal"
Jul 30 10:48:14.984509 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:14.984495    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-243.ec2.internal\" not found" node="ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.050839 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:15.050818    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:15.052979 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.052967    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db8ff15800a49eb6cf75db11d8895b81-config\") pod \"kube-apiserver-proxy-ip-10-0-128-243.ec2.internal\" (UID: \"db8ff15800a49eb6cf75db11d8895b81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.053042 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.052989    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d15aaab8d00e7896dae80fe2b70d2f3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal\" (UID: \"d15aaab8d00e7896dae80fe2b70d2f3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.053042 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.053006    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d15aaab8d00e7896dae80fe2b70d2f3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal\" (UID: \"d15aaab8d00e7896dae80fe2b70d2f3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.053133 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.053044    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db8ff15800a49eb6cf75db11d8895b81-config\") pod \"kube-apiserver-proxy-ip-10-0-128-243.ec2.internal\" (UID: \"db8ff15800a49eb6cf75db11d8895b81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.151198 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:15.151138    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:15.153295 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.153282    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d15aaab8d00e7896dae80fe2b70d2f3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal\" (UID: \"d15aaab8d00e7896dae80fe2b70d2f3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.153344 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.153301    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d15aaab8d00e7896dae80fe2b70d2f3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal\" (UID: \"d15aaab8d00e7896dae80fe2b70d2f3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.153344 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.153336    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d15aaab8d00e7896dae80fe2b70d2f3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal\" (UID: \"d15aaab8d00e7896dae80fe2b70d2f3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.153432 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.153397    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d15aaab8d00e7896dae80fe2b70d2f3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal\" (UID: \"d15aaab8d00e7896dae80fe2b70d2f3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.251449 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:15.251408    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:15.282680 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.282665    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.288510 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.288493    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.351508 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:15.351482    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:15.451724 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:15.451662    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:15.552091 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:15.552065    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:15.598454 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.598434    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 30 10:48:15.598917 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.598575    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 10:48:15.652245 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:15.652222    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-243.ec2.internal\" not found"
Jul 30 10:48:15.685304 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.685278    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 10:43:14 +0000 UTC" deadline="2028-01-24 05:24:19.710570805 +0000 UTC"
Jul 30 10:48:15.685304 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.685304    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13026h36m4.025271037s"
Jul 30 10:48:15.707647 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.707600    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 10:48:15.751053 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.751032    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.765977 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.765962    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 10:48:15.767519 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.767506    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal"
Jul 30 10:48:15.768841 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:15.768820    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15aaab8d00e7896dae80fe2b70d2f3b.slice/crio-29d3978ae92614e028c123ce1f31426eb64bf5f456821a41cd4cd2fd9b56a4be WatchSource:0}: Error finding container 29d3978ae92614e028c123ce1f31426eb64bf5f456821a41cd4cd2fd9b56a4be: Status 404 returned error can't find the container with id 29d3978ae92614e028c123ce1f31426eb64bf5f456821a41cd4cd2fd9b56a4be
Jul 30 10:48:15.769210 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:15.769197    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8ff15800a49eb6cf75db11d8895b81.slice/crio-dfc1a114773d8590d8c710829760b2008d9cfc64a8525cc45d6ea221aca20cf7 WatchSource:0}: Error finding container dfc1a114773d8590d8c710829760b2008d9cfc64a8525cc45d6ea221aca20cf7: Status 404 returned error can't find the container with id dfc1a114773d8590d8c710829760b2008d9cfc64a8525cc45d6ea221aca20cf7
Jul 30 10:48:15.773389 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.773374    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 10:48:15.803321 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.803307    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 10:48:15.852218 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.852182    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal" event={"ID":"db8ff15800a49eb6cf75db11d8895b81","Type":"ContainerStarted","Data":"dfc1a114773d8590d8c710829760b2008d9cfc64a8525cc45d6ea221aca20cf7"}
Jul 30 10:48:15.853137 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.853116    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal" event={"ID":"d15aaab8d00e7896dae80fe2b70d2f3b","Type":"ContainerStarted","Data":"29d3978ae92614e028c123ce1f31426eb64bf5f456821a41cd4cd2fd9b56a4be"}
Jul 30 10:48:15.907227 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:15.907200    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 10:48:16.157554 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.157482    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 10:48:16.629979 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.629905    2571 apiserver.go:52] "Watching apiserver"
Jul 30 10:48:16.638910 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.638881    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 30 10:48:16.640608 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.640575    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-grqcq","openshift-image-registry/node-ca-wvnjw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal","openshift-multus/multus-additional-cni-plugins-mqfr5","openshift-multus/multus-fl6ff","openshift-multus/network-metrics-daemon-5qwx4","kube-system/konnectivity-agent-9t4zj","kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf","openshift-network-diagnostics/network-check-target-x72dc","openshift-network-operator/iptables-alerter-jrkr8","openshift-ovn-kubernetes/ovnkube-node-qltbr","kube-system/global-pull-secret-syncer-cbrck","openshift-cluster-node-tuning-operator/tuned-4bfpq"]
Jul 30 10:48:16.642435 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.642406    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:16.643734 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.643714    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:16.643837 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.643792    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:16.644950 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.644927    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.645883 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.645861    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 30 10:48:16.646372 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.646240    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.646372 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.646291    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 30 10:48:16.646504 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.646384    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b64dz\""
Jul 30 10:48:16.647489 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.647470    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.648895 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.648878    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:16.648980 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.648953    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:16.650070 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.650056    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.650263 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.650244    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 30 10:48:16.650506 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.650491    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-shpkg\""
Jul 30 10:48:16.650729 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.650688    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 30 10:48:16.651627 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.651606    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:16.652613 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.652595    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:16.653844 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.653825    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:16.653934 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.653903    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:16.655744 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.655324    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.657544 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.657522    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.659520 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659495    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-run-k8s-cni-cncf-io\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.659598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659532    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-var-lib-cni-bin\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.659598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659556    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-device-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.659598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659578    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-system-cni-dir\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.659758 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659601    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.659758 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659622    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-os-release\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.659758 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659648    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-run-netns\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.659758 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659671    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-multus-conf-dir\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.659758 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659693    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-kubernetes\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.659758 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-var-lib-kubelet\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.659758 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659738    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtqh\" (UniqueName: \"kubernetes.io/projected/e881cdff-855a-4f17-9552-320d9b7a7daf-kube-api-access-nwtqh\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.659996 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659778    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-cnibin\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.659996 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659816    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/444591cd-6538-4674-a92e-783563d56021-cni-binary-copy\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.659996 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659856    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bb7436-3324-44f3-8e91-c4a972c6c38f-tmp-dir\") pod \"node-resolver-grqcq\" (UID: \"90bb7436-3324-44f3-8e91-c4a972c6c38f\") " pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:16.659996 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659958    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2a27b06-fcf7-4cbf-a646-ee312bada871-agent-certs\") pod \"konnectivity-agent-9t4zj\" (UID: \"d2a27b06-fcf7-4cbf-a646-ee312bada871\") " pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:16.659996 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.659984    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cqxj\" (UniqueName: \"kubernetes.io/projected/5df252dc-1dc8-40e4-a5f9-465fc2ee5819-kube-api-access-5cqxj\") pod \"node-ca-wvnjw\" (UID: \"5df252dc-1dc8-40e4-a5f9-465fc2ee5819\") " pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:16.660205 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660018    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-sysctl-conf\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.660205 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660043    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.660205 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660079    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgmvm\" (UniqueName: \"kubernetes.io/projected/94ab27bc-722b-4c3e-84bb-97dff2669a57-kube-api-access-vgmvm\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.660205 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660149    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-modprobe-d\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.660205 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e881cdff-855a-4f17-9552-320d9b7a7daf-tmp\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.660399 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660214    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-system-cni-dir\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.660399 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660238    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/444591cd-6538-4674-a92e-783563d56021-multus-daemon-config\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.660399 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660263    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7q8d\" (UniqueName: \"kubernetes.io/projected/444591cd-6538-4674-a92e-783563d56021-kube-api-access-j7q8d\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.660399 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660288    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:16.660399 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660326    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-kubelet-config\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:16.660399 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660348    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-run\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.660399 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660369    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-cnibin\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.660399 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-socket-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.660666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5df252dc-1dc8-40e4-a5f9-465fc2ee5819-host\") pod \"node-ca-wvnjw\" (UID: \"5df252dc-1dc8-40e4-a5f9-465fc2ee5819\") " pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:16.660666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5df252dc-1dc8-40e4-a5f9-465fc2ee5819-serviceca\") pod \"node-ca-wvnjw\" (UID: \"5df252dc-1dc8-40e4-a5f9-465fc2ee5819\") " pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:16.660666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660460    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-systemd\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.660666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660483    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-run-multus-certs\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.660666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660515    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-lib-modules\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.660666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v727d\" (UniqueName: \"kubernetes.io/projected/ab429725-9ff8-4ee4-97e8-e5c107e34235-kube-api-access-v727d\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.660666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660562    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mbjj\" (UniqueName: \"kubernetes.io/projected/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-kube-api-access-8mbjj\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:16.660666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660585    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-etc-selinux\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.660666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-sys\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.660666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-multus-cni-dir\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660671    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-var-lib-cni-multus\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660695    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-etc-kubernetes\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660722    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90bb7436-3324-44f3-8e91-c4a972c6c38f-hosts-file\") pod \"node-resolver-grqcq\" (UID: \"90bb7436-3324-44f3-8e91-c4a972c6c38f\") " pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660760    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab429725-9ff8-4ee4-97e8-e5c107e34235-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660787    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660810    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-sysconfig\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660828    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-os-release\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660847    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-registration-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660880    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xcd\" (UniqueName: \"kubernetes.io/projected/90bb7436-3324-44f3-8e91-c4a972c6c38f-kube-api-access-s8xcd\") pod \"node-resolver-grqcq\" (UID: \"90bb7436-3324-44f3-8e91-c4a972c6c38f\") " pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-dbus\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660914    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-host\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660934    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab429725-9ff8-4ee4-97e8-e5c107e34235-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660963    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-sysctl-d\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.660982    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-tuned\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.661002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-var-lib-kubelet\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.661139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.661036    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-hostroot\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.662032 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.661060    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-sys-fs\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.662032 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.661110    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab429725-9ff8-4ee4-97e8-e5c107e34235-cni-binary-copy\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.662032 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.661135    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2a27b06-fcf7-4cbf-a646-ee312bada871-konnectivity-ca\") pod \"konnectivity-agent-9t4zj\" (UID: \"d2a27b06-fcf7-4cbf-a646-ee312bada871\") " pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:16.662032 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.661172    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-multus-socket-dir-parent\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.665517 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.665494    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 30 10:48:16.666006 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.665988    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 30 10:48:16.666267 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.666246    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pg7ns\""
Jul 30 10:48:16.669177 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.667779    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 30 10:48:16.669177 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.668112    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 30 10:48:16.669177 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.668162    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 30 10:48:16.669177 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.668406    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 30 10:48:16.669177 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.668493    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 30 10:48:16.669177 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.668769    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 30 10:48:16.669177 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.668925    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 30 10:48:16.669177 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.669164    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.670647    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.670818    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.670968    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671008    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671123    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671139    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671280    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671367    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671381    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671397    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7xsrz\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671462    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pb95p\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671280    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 30 10:48:16.671621 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671465    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m7j5j\""
Jul 30 10:48:16.672296 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671704    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 30 10:48:16.672296 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671707    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 30 10:48:16.672296 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671830    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 30 10:48:16.672296 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.671894    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zkcqp\""
Jul 30 10:48:16.672665 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.672646    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dhsm7\""
Jul 30 10:48:16.673266 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.673070    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 30 10:48:16.673266 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.673115    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rgfwk\""
Jul 30 10:48:16.726316 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.726296    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 10:48:16.752996 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.752978    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 30 10:48:16.761483 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761465    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-sysconfig\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.761580 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761495    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-os-release\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.761580 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761516    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-registration-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.761580 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761535    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-cni-netd\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.761580 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761551    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xcd\" (UniqueName: \"kubernetes.io/projected/90bb7436-3324-44f3-8e91-c4a972c6c38f-kube-api-access-s8xcd\") pod \"node-resolver-grqcq\" (UID: \"90bb7436-3324-44f3-8e91-c4a972c6c38f\") " pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:16.761580 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4l77\" (UniqueName: \"kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77\") pod \"network-check-target-x72dc\" (UID: \"18109ae3-f141-4b3a-aba4-37f3a8d82cdc\") " pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761597    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-sysconfig\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761601    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-run-netns\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-registration-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761610    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-os-release\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761648    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-run-ovn\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-run-ovn-kubernetes\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761704    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8ccc70b-7123-458f-aeb5-d0f954af3515-env-overrides\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761733    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-dbus\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761756    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-host\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761799    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab429725-9ff8-4ee4-97e8-e5c107e34235-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761823    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-sysctl-d\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761856    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-tuned\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761861    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-dbus\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761879    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-var-lib-kubelet\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761897    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-host\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761902    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-hostroot\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.761935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761942    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-hostroot\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761948    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-sys-fs\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761956    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-sysctl-d\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761979    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab429725-9ff8-4ee4-97e8-e5c107e34235-cni-binary-copy\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.761995    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-sys-fs\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762008    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2a27b06-fcf7-4cbf-a646-ee312bada871-konnectivity-ca\") pod \"konnectivity-agent-9t4zj\" (UID: \"d2a27b06-fcf7-4cbf-a646-ee312bada871\") " pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762013    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-var-lib-kubelet\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762038    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-cni-bin\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762063    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjf4d\" (UniqueName: \"kubernetes.io/projected/9480f1c4-db3e-49b2-b4e3-a95469309c7b-kube-api-access-vjf4d\") pod \"iptables-alerter-jrkr8\" (UID: \"9480f1c4-db3e-49b2-b4e3-a95469309c7b\") " pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762089    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-multus-socket-dir-parent\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762142    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-run-k8s-cni-cncf-io\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762167    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-multus-socket-dir-parent\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762173    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-var-lib-cni-bin\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762203    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-device-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762210    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-run-k8s-cni-cncf-io\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-systemd-units\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762252    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-var-lib-cni-bin\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.762737 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762261    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8ccc70b-7123-458f-aeb5-d0f954af3515-ovn-node-metrics-cert\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762295    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-system-cni-dir\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762273    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762322    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-device-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762323    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762360    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-os-release\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762384    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-run-netns\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-multus-conf-dir\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-kubernetes\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-var-lib-kubelet\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762462    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762479    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtqh\" (UniqueName: \"kubernetes.io/projected/e881cdff-855a-4f17-9552-320d9b7a7daf-kube-api-access-nwtqh\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-cnibin\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762565    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-cnibin\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762576    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-system-cni-dir\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762591    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-var-lib-kubelet\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762630    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-run-netns\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762638    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-multus-conf-dir\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.763598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762642    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2a27b06-fcf7-4cbf-a646-ee312bada871-konnectivity-ca\") pod \"konnectivity-agent-9t4zj\" (UID: \"d2a27b06-fcf7-4cbf-a646-ee312bada871\") " pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762648    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab429725-9ff8-4ee4-97e8-e5c107e34235-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762682    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-kubernetes\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762642    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab429725-9ff8-4ee4-97e8-e5c107e34235-cni-binary-copy\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762706    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/444591cd-6538-4674-a92e-783563d56021-cni-binary-copy\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762744    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bb7436-3324-44f3-8e91-c4a972c6c38f-tmp-dir\") pod \"node-resolver-grqcq\" (UID: \"90bb7436-3324-44f3-8e91-c4a972c6c38f\") " pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762767    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-os-release\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762776    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2a27b06-fcf7-4cbf-a646-ee312bada871-agent-certs\") pod \"konnectivity-agent-9t4zj\" (UID: \"d2a27b06-fcf7-4cbf-a646-ee312bada871\") " pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762866    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cqxj\" (UniqueName: \"kubernetes.io/projected/5df252dc-1dc8-40e4-a5f9-465fc2ee5819-kube-api-access-5cqxj\") pod \"node-ca-wvnjw\" (UID: \"5df252dc-1dc8-40e4-a5f9-465fc2ee5819\") " pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762908    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-sysctl-conf\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762961    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgmvm\" (UniqueName: \"kubernetes.io/projected/94ab27bc-722b-4c3e-84bb-97dff2669a57-kube-api-access-vgmvm\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.762989    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-run-openvswitch\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763014    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-log-socket\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763042    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763046    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bb7436-3324-44f3-8e91-c4a972c6c38f-tmp-dir\") pod \"node-resolver-grqcq\" (UID: \"90bb7436-3324-44f3-8e91-c4a972c6c38f\") " pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763071    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9480f1c4-db3e-49b2-b4e3-a95469309c7b-iptables-alerter-script\") pod \"iptables-alerter-jrkr8\" (UID: \"9480f1c4-db3e-49b2-b4e3-a95469309c7b\") " pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.764481 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763117    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-modprobe-d\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763167    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e881cdff-855a-4f17-9552-320d9b7a7daf-tmp\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-system-cni-dir\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763215    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-sysctl-conf\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763259    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/444591cd-6538-4674-a92e-783563d56021-multus-daemon-config\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763279    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/444591cd-6538-4674-a92e-783563d56021-cni-binary-copy\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763286    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8ccc70b-7123-458f-aeb5-d0f954af3515-ovnkube-config\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763299    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763261    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-modprobe-d\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763308    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9480f1c4-db3e-49b2-b4e3-a95469309c7b-host-slash\") pod \"iptables-alerter-jrkr8\" (UID: \"9480f1c4-db3e-49b2-b4e3-a95469309c7b\") " pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763350    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7q8d\" (UniqueName: \"kubernetes.io/projected/444591cd-6538-4674-a92e-783563d56021-kube-api-access-j7q8d\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763360    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-system-cni-dir\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763535    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-run-systemd\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.763582    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-var-lib-openvswitch\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763640    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-node-log\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.765346 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.763728    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs podName:c534fc6d-98cb-4a6e-b36a-1b8d6a61de79 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:17.263696787 +0000 UTC m=+2.984581041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs") pod "network-metrics-daemon-5qwx4" (UID: "c534fc6d-98cb-4a6e-b36a-1b8d6a61de79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763749    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-kubelet-config\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763774    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-run\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763797    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-cnibin\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763820    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-socket-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763851    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5df252dc-1dc8-40e4-a5f9-465fc2ee5819-host\") pod \"node-ca-wvnjw\" (UID: \"5df252dc-1dc8-40e4-a5f9-465fc2ee5819\") " pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763858    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-kubelet-config\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5df252dc-1dc8-40e4-a5f9-465fc2ee5819-serviceca\") pod \"node-ca-wvnjw\" (UID: \"5df252dc-1dc8-40e4-a5f9-465fc2ee5819\") " pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-etc-openvswitch\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763926    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-systemd\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763917    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab429725-9ff8-4ee4-97e8-e5c107e34235-cnibin\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763938    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5df252dc-1dc8-40e4-a5f9-465fc2ee5819-host\") pod \"node-ca-wvnjw\" (UID: \"5df252dc-1dc8-40e4-a5f9-465fc2ee5819\") " pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8ccc70b-7123-458f-aeb5-d0f954af3515-ovnkube-script-lib\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763987    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-run\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.763991    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-run-multus-certs\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764024    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-kubelet\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-slash\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.766004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764028    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-run-multus-certs\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764086    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlg7v\" (UniqueName: \"kubernetes.io/projected/c8ccc70b-7123-458f-aeb5-d0f954af3515-kube-api-access-dlg7v\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-lib-modules\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764163    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-socket-dir\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764161    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v727d\" (UniqueName: \"kubernetes.io/projected/ab429725-9ff8-4ee4-97e8-e5c107e34235-kube-api-access-v727d\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764218    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mbjj\" (UniqueName: \"kubernetes.io/projected/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-kube-api-access-8mbjj\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764231    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-systemd\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764306    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/444591cd-6538-4674-a92e-783563d56021-multus-daemon-config\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764328    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-lib-modules\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764367    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5df252dc-1dc8-40e4-a5f9-465fc2ee5819-serviceca\") pod \"node-ca-wvnjw\" (UID: \"5df252dc-1dc8-40e4-a5f9-465fc2ee5819\") " pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764246    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-etc-selinux\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764565    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-sys\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764593    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/94ab27bc-722b-4c3e-84bb-97dff2669a57-etc-selinux\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764602    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-multus-cni-dir\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-var-lib-cni-multus\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764646    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e881cdff-855a-4f17-9552-320d9b7a7daf-sys\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-etc-kubernetes\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764682    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-multus-cni-dir\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.766615 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764694    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-host-var-lib-cni-multus\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764724    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90bb7436-3324-44f3-8e91-c4a972c6c38f-hosts-file\") pod \"node-resolver-grqcq\" (UID: \"90bb7436-3324-44f3-8e91-c4a972c6c38f\") " pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764749    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/444591cd-6538-4674-a92e-783563d56021-etc-kubernetes\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764752    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab429725-9ff8-4ee4-97e8-e5c107e34235-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.764812    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90bb7436-3324-44f3-8e91-c4a972c6c38f-hosts-file\") pod \"node-resolver-grqcq\" (UID: \"90bb7436-3324-44f3-8e91-c4a972c6c38f\") " pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.764891    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.764937    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret podName:20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d nodeName:}" failed. No retries permitted until 2026-07-30 10:48:17.264925683 +0000 UTC m=+2.985809924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret") pod "global-pull-secret-syncer-cbrck" (UID: "20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d") : object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.765260    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab429725-9ff8-4ee4-97e8-e5c107e34235-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.765841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e881cdff-855a-4f17-9552-320d9b7a7daf-etc-tuned\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.766020    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2a27b06-fcf7-4cbf-a646-ee312bada871-agent-certs\") pod \"konnectivity-agent-9t4zj\" (UID: \"d2a27b06-fcf7-4cbf-a646-ee312bada871\") " pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:16.767201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.766282    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e881cdff-855a-4f17-9552-320d9b7a7daf-tmp\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.777684 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.777662    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mlg5j"
Jul 30 10:48:16.781260 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.781235    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xcd\" (UniqueName: \"kubernetes.io/projected/90bb7436-3324-44f3-8e91-c4a972c6c38f-kube-api-access-s8xcd\") pod \"node-resolver-grqcq\" (UID: \"90bb7436-3324-44f3-8e91-c4a972c6c38f\") " pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:16.790675 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.790652    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cqxj\" (UniqueName: \"kubernetes.io/projected/5df252dc-1dc8-40e4-a5f9-465fc2ee5819-kube-api-access-5cqxj\") pod \"node-ca-wvnjw\" (UID: \"5df252dc-1dc8-40e4-a5f9-465fc2ee5819\") " pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:16.794700 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.794677    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtqh\" (UniqueName: \"kubernetes.io/projected/e881cdff-855a-4f17-9552-320d9b7a7daf-kube-api-access-nwtqh\") pod \"tuned-4bfpq\" (UID: \"e881cdff-855a-4f17-9552-320d9b7a7daf\") " pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.794975 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.794956    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v727d\" (UniqueName: \"kubernetes.io/projected/ab429725-9ff8-4ee4-97e8-e5c107e34235-kube-api-access-v727d\") pod \"multus-additional-cni-plugins-mqfr5\" (UID: \"ab429725-9ff8-4ee4-97e8-e5c107e34235\") " pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.795051 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.794991    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mbjj\" (UniqueName: \"kubernetes.io/projected/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-kube-api-access-8mbjj\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:16.795413 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.795393    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7q8d\" (UniqueName: \"kubernetes.io/projected/444591cd-6538-4674-a92e-783563d56021-kube-api-access-j7q8d\") pod \"multus-fl6ff\" (UID: \"444591cd-6538-4674-a92e-783563d56021\") " pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:16.795486 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.795400    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgmvm\" (UniqueName: \"kubernetes.io/projected/94ab27bc-722b-4c3e-84bb-97dff2669a57-kube-api-access-vgmvm\") pod \"aws-ebs-csi-driver-node-wsbkf\" (UID: \"94ab27bc-722b-4c3e-84bb-97dff2669a57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:16.796622 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.796605    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mlg5j"
Jul 30 10:48:16.865729 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865693    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-cni-netd\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.865887 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l77\" (UniqueName: \"kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77\") pod \"network-check-target-x72dc\" (UID: \"18109ae3-f141-4b3a-aba4-37f3a8d82cdc\") " pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:16.865887 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865773    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-run-netns\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.865887 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865796    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-run-ovn\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.865887 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865814    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-cni-netd\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.865887 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865824    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-run-ovn-kubernetes\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.865887 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865858    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8ccc70b-7123-458f-aeb5-d0f954af3515-env-overrides\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865896    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-cni-bin\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865899    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-run-ovn\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865921    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjf4d\" (UniqueName: \"kubernetes.io/projected/9480f1c4-db3e-49b2-b4e3-a95469309c7b-kube-api-access-vjf4d\") pod \"iptables-alerter-jrkr8\" (UID: \"9480f1c4-db3e-49b2-b4e3-a95469309c7b\") " pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865860    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-run-ovn-kubernetes\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865897    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-run-netns\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865954    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-systemd-units\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865957    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-cni-bin\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865986    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-systemd-units\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.865994    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8ccc70b-7123-458f-aeb5-d0f954af3515-ovn-node-metrics-cert\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-run-openvswitch\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866067    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-log-socket\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866092    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9480f1c4-db3e-49b2-b4e3-a95469309c7b-iptables-alerter-script\") pod \"iptables-alerter-jrkr8\" (UID: \"9480f1c4-db3e-49b2-b4e3-a95469309c7b\") " pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866145    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8ccc70b-7123-458f-aeb5-d0f954af3515-ovnkube-config\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866162    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9480f1c4-db3e-49b2-b4e3-a95469309c7b-host-slash\") pod \"iptables-alerter-jrkr8\" (UID: \"9480f1c4-db3e-49b2-b4e3-a95469309c7b\") " pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866198    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-run-systemd\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866221    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-var-lib-openvswitch\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866236    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-node-log\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866241    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866266    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-etc-openvswitch\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866286    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8ccc70b-7123-458f-aeb5-d0f954af3515-ovnkube-script-lib\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866294    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-run-openvswitch\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866307    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-kubelet\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866331    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-log-socket\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866334    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-slash\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866340    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8ccc70b-7123-458f-aeb5-d0f954af3515-env-overrides\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866361    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-slash\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866388    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-var-lib-openvswitch\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866403    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-node-log\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866421    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9480f1c4-db3e-49b2-b4e3-a95469309c7b-host-slash\") pod \"iptables-alerter-jrkr8\" (UID: \"9480f1c4-db3e-49b2-b4e3-a95469309c7b\") " pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866425    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlg7v\" (UniqueName: \"kubernetes.io/projected/c8ccc70b-7123-458f-aeb5-d0f954af3515-kube-api-access-dlg7v\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866478    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-etc-openvswitch\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866516    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-run-systemd\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866716    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8ccc70b-7123-458f-aeb5-d0f954af3515-host-kubelet\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.866980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866868    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8ccc70b-7123-458f-aeb5-d0f954af3515-ovnkube-config\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.867708 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866948    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8ccc70b-7123-458f-aeb5-d0f954af3515-ovnkube-script-lib\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.867708 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.866974    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9480f1c4-db3e-49b2-b4e3-a95469309c7b-iptables-alerter-script\") pod \"iptables-alerter-jrkr8\" (UID: \"9480f1c4-db3e-49b2-b4e3-a95469309c7b\") " pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.868562 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.868513    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8ccc70b-7123-458f-aeb5-d0f954af3515-ovn-node-metrics-cert\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.882992 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.882938    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 10:48:16.882992 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.882963    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 10:48:16.882992 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.882979    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c4l77 for pod openshift-network-diagnostics/network-check-target-x72dc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:16.883208 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:16.883038    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77 podName:18109ae3-f141-4b3a-aba4-37f3a8d82cdc nodeName:}" failed. No retries permitted until 2026-07-30 10:48:17.383023569 +0000 UTC m=+3.103907808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c4l77" (UniqueName: "kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77") pod "network-check-target-x72dc" (UID: "18109ae3-f141-4b3a-aba4-37f3a8d82cdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:16.899166 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.899133    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlg7v\" (UniqueName: \"kubernetes.io/projected/c8ccc70b-7123-458f-aeb5-d0f954af3515-kube-api-access-dlg7v\") pod \"ovnkube-node-qltbr\" (UID: \"c8ccc70b-7123-458f-aeb5-d0f954af3515\") " pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:16.899372 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.899351    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjf4d\" (UniqueName: \"kubernetes.io/projected/9480f1c4-db3e-49b2-b4e3-a95469309c7b-kube-api-access-vjf4d\") pod \"iptables-alerter-jrkr8\" (UID: \"9480f1c4-db3e-49b2-b4e3-a95469309c7b\") " pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.954806 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.954775    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:16.964614 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.964593    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4bfpq"
Jul 30 10:48:16.983566 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.983546    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jrkr8"
Jul 30 10:48:16.989168 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.989143    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mqfr5"
Jul 30 10:48:16.995610 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:16.995594    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fl6ff"
Jul 30 10:48:17.004198 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.004177    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf"
Jul 30 10:48:17.011699 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.011682    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-grqcq"
Jul 30 10:48:17.018208 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.018184    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wvnjw"
Jul 30 10:48:17.023882 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.023863    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:17.066080 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.066057    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 10:48:17.270604 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.270512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:17.270604 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.270577    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:17.270788 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:17.270681    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:17.270788 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:17.270739    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs podName:c534fc6d-98cb-4a6e-b36a-1b8d6a61de79 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:18.270724377 +0000 UTC m=+3.991608619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs") pod "network-metrics-daemon-5qwx4" (UID: "c534fc6d-98cb-4a6e-b36a-1b8d6a61de79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:17.270788 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:17.270745    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:17.270905 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:17.270810    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret podName:20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d nodeName:}" failed. No retries permitted until 2026-07-30 10:48:18.270790379 +0000 UTC m=+3.991674636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret") pod "global-pull-secret-syncer-cbrck" (UID: "20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d") : object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:17.473085 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.473044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l77\" (UniqueName: \"kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77\") pod \"network-check-target-x72dc\" (UID: \"18109ae3-f141-4b3a-aba4-37f3a8d82cdc\") " pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:17.473245 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:17.473190    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 10:48:17.473245 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:17.473207    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 10:48:17.473245 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:17.473216    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c4l77 for pod openshift-network-diagnostics/network-check-target-x72dc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:17.473363 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:17.473265    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77 podName:18109ae3-f141-4b3a-aba4-37f3a8d82cdc nodeName:}" failed. No retries permitted until 2026-07-30 10:48:18.473251797 +0000 UTC m=+4.194136036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4l77" (UniqueName: "kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77") pod "network-check-target-x72dc" (UID: "18109ae3-f141-4b3a-aba4-37f3a8d82cdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:17.592485 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:17.592371    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ccc70b_7123_458f_aeb5_d0f954af3515.slice/crio-b05787bc8053096e212eeda4687c23da37133dd481f6fb3ffc24ff8a19b51af9 WatchSource:0}: Error finding container b05787bc8053096e212eeda4687c23da37133dd481f6fb3ffc24ff8a19b51af9: Status 404 returned error can't find the container with id b05787bc8053096e212eeda4687c23da37133dd481f6fb3ffc24ff8a19b51af9
Jul 30 10:48:17.593624 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:17.593596    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ab27bc_722b_4c3e_84bb_97dff2669a57.slice/crio-35e9e5aeea73a561dcbe26566a12ac6ef381fcb4aa56e4d0b7b0f36d49d9e056 WatchSource:0}: Error finding container 35e9e5aeea73a561dcbe26566a12ac6ef381fcb4aa56e4d0b7b0f36d49d9e056: Status 404 returned error can't find the container with id 35e9e5aeea73a561dcbe26566a12ac6ef381fcb4aa56e4d0b7b0f36d49d9e056
Jul 30 10:48:17.594220 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:17.594134    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a27b06_fcf7_4cbf_a646_ee312bada871.slice/crio-df5c192af7ac0ffd3921d0345936c09b65ccfab46b4178aeddea4457a578ac1e WatchSource:0}: Error finding container df5c192af7ac0ffd3921d0345936c09b65ccfab46b4178aeddea4457a578ac1e: Status 404 returned error can't find the container with id df5c192af7ac0ffd3921d0345936c09b65ccfab46b4178aeddea4457a578ac1e
Jul 30 10:48:17.594843 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:17.594811    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9480f1c4_db3e_49b2_b4e3_a95469309c7b.slice/crio-8c54327ca2268364e24213cf20ae4529906372b410a0c8ed4a3ae01a3908898d WatchSource:0}: Error finding container 8c54327ca2268364e24213cf20ae4529906372b410a0c8ed4a3ae01a3908898d: Status 404 returned error can't find the container with id 8c54327ca2268364e24213cf20ae4529906372b410a0c8ed4a3ae01a3908898d
Jul 30 10:48:17.596614 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:17.596475    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode881cdff_855a_4f17_9552_320d9b7a7daf.slice/crio-45e2cc3452b9aa8c458ec638e34792882ab77fd0460d2c7ce3dd552e6fa5c911 WatchSource:0}: Error finding container 45e2cc3452b9aa8c458ec638e34792882ab77fd0460d2c7ce3dd552e6fa5c911: Status 404 returned error can't find the container with id 45e2cc3452b9aa8c458ec638e34792882ab77fd0460d2c7ce3dd552e6fa5c911
Jul 30 10:48:17.598943 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:17.598880    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444591cd_6538_4674_a92e_783563d56021.slice/crio-16cb7f12adce27b5d8a6f93717d5a02e057901c75a478ef7efea8511f6f7a441 WatchSource:0}: Error finding container 16cb7f12adce27b5d8a6f93717d5a02e057901c75a478ef7efea8511f6f7a441: Status 404 returned error can't find the container with id 16cb7f12adce27b5d8a6f93717d5a02e057901c75a478ef7efea8511f6f7a441
Jul 30 10:48:17.599740 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:17.599717    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bb7436_3324_44f3_8e91_c4a972c6c38f.slice/crio-0db4f2bd89fc86d24aad21a07cc75e7164f3114560be61ae333efbda1f38b49b WatchSource:0}: Error finding container 0db4f2bd89fc86d24aad21a07cc75e7164f3114560be61ae333efbda1f38b49b: Status 404 returned error can't find the container with id 0db4f2bd89fc86d24aad21a07cc75e7164f3114560be61ae333efbda1f38b49b
Jul 30 10:48:17.601157 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:17.601132    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df252dc_1dc8_40e4_a5f9_465fc2ee5819.slice/crio-6c1640ab1453ea5f5829c99e36f9d17d8e81d970fc2d34c82c404d1fff68085c WatchSource:0}: Error finding container 6c1640ab1453ea5f5829c99e36f9d17d8e81d970fc2d34c82c404d1fff68085c: Status 404 returned error can't find the container with id 6c1640ab1453ea5f5829c99e36f9d17d8e81d970fc2d34c82c404d1fff68085c
Jul 30 10:48:17.602351 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:17.601968    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab429725_9ff8_4ee4_97e8_e5c107e34235.slice/crio-327cb067e7c84156596fef18d4c471a5f1b6fe5786b043e8a7eb47588c18e3a8 WatchSource:0}: Error finding container 327cb067e7c84156596fef18d4c471a5f1b6fe5786b043e8a7eb47588c18e3a8: Status 404 returned error can't find the container with id 327cb067e7c84156596fef18d4c471a5f1b6fe5786b043e8a7eb47588c18e3a8
Jul 30 10:48:17.799492 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.799312    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 10:43:16 +0000 UTC" deadline="2028-03-31 09:43:18.728504635 +0000 UTC"
Jul 30 10:48:17.799492 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.799487    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14638h55m0.929021422s"
Jul 30 10:48:17.849612 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.849589    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:17.849744 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:17.849721    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:17.849814 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.849770    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:17.849868 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:17.849854    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:17.858878 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.858710    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal" event={"ID":"db8ff15800a49eb6cf75db11d8895b81","Type":"ContainerStarted","Data":"a2716bcb6cd38e73eacd60e7dd17e216dcc53f694d6c22c6001fa0156bed76fc"}
Jul 30 10:48:17.860142 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.860093    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfr5" event={"ID":"ab429725-9ff8-4ee4-97e8-e5c107e34235","Type":"ContainerStarted","Data":"327cb067e7c84156596fef18d4c471a5f1b6fe5786b043e8a7eb47588c18e3a8"}
Jul 30 10:48:17.861444 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.861424    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wvnjw" event={"ID":"5df252dc-1dc8-40e4-a5f9-465fc2ee5819","Type":"ContainerStarted","Data":"6c1640ab1453ea5f5829c99e36f9d17d8e81d970fc2d34c82c404d1fff68085c"}
Jul 30 10:48:17.862438 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.862418    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-grqcq" event={"ID":"90bb7436-3324-44f3-8e91-c4a972c6c38f","Type":"ContainerStarted","Data":"0db4f2bd89fc86d24aad21a07cc75e7164f3114560be61ae333efbda1f38b49b"}
Jul 30 10:48:17.863319 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.863292    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fl6ff" event={"ID":"444591cd-6538-4674-a92e-783563d56021","Type":"ContainerStarted","Data":"16cb7f12adce27b5d8a6f93717d5a02e057901c75a478ef7efea8511f6f7a441"}
Jul 30 10:48:17.864201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.864184    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4bfpq" event={"ID":"e881cdff-855a-4f17-9552-320d9b7a7daf","Type":"ContainerStarted","Data":"45e2cc3452b9aa8c458ec638e34792882ab77fd0460d2c7ce3dd552e6fa5c911"}
Jul 30 10:48:17.865217 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.865197    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jrkr8" event={"ID":"9480f1c4-db3e-49b2-b4e3-a95469309c7b","Type":"ContainerStarted","Data":"8c54327ca2268364e24213cf20ae4529906372b410a0c8ed4a3ae01a3908898d"}
Jul 30 10:48:17.866535 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.866515    2571 generic.go:356] "Generic (PLEG): container finished" podID="d15aaab8d00e7896dae80fe2b70d2f3b" containerID="51ac4d8ecc43cc4f79f8c340a274c6e722edd76a0a80565f275b33a3eaa2d194" exitCode=0
Jul 30 10:48:17.866665 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.866591    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal" event={"ID":"d15aaab8d00e7896dae80fe2b70d2f3b","Type":"ContainerDied","Data":"51ac4d8ecc43cc4f79f8c340a274c6e722edd76a0a80565f275b33a3eaa2d194"}
Jul 30 10:48:17.867616 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.867598    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9t4zj" event={"ID":"d2a27b06-fcf7-4cbf-a646-ee312bada871","Type":"ContainerStarted","Data":"df5c192af7ac0ffd3921d0345936c09b65ccfab46b4178aeddea4457a578ac1e"}
Jul 30 10:48:17.868657 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.868638    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf" event={"ID":"94ab27bc-722b-4c3e-84bb-97dff2669a57","Type":"ContainerStarted","Data":"35e9e5aeea73a561dcbe26566a12ac6ef381fcb4aa56e4d0b7b0f36d49d9e056"}
Jul 30 10:48:17.869593 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.869575    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr" event={"ID":"c8ccc70b-7123-458f-aeb5-d0f954af3515","Type":"ContainerStarted","Data":"b05787bc8053096e212eeda4687c23da37133dd481f6fb3ffc24ff8a19b51af9"}
Jul 30 10:48:17.888772 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:17.888732    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal" podStartSLOduration=2.888719237 podStartE2EDuration="2.888719237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:17.88818645 +0000 UTC m=+3.609070712" watchObservedRunningTime="2026-07-30 10:48:17.888719237 +0000 UTC m=+3.609603498"
Jul 30 10:48:18.279533 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:18.279444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:18.279686 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:18.279548    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:18.279686 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:18.279612    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:18.279686 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:18.279660    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:18.279868 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:18.279723    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs podName:c534fc6d-98cb-4a6e-b36a-1b8d6a61de79 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:20.27970089 +0000 UTC m=+6.000585129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs") pod "network-metrics-daemon-5qwx4" (UID: "c534fc6d-98cb-4a6e-b36a-1b8d6a61de79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:18.279868 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:18.279756    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret podName:20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d nodeName:}" failed. No retries permitted until 2026-07-30 10:48:20.279732881 +0000 UTC m=+6.000617134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret") pod "global-pull-secret-syncer-cbrck" (UID: "20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d") : object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:18.482214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:18.482167    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l77\" (UniqueName: \"kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77\") pod \"network-check-target-x72dc\" (UID: \"18109ae3-f141-4b3a-aba4-37f3a8d82cdc\") " pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:18.482379 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:18.482328    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 10:48:18.482379 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:18.482351    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 10:48:18.482379 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:18.482363    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c4l77 for pod openshift-network-diagnostics/network-check-target-x72dc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:18.482703 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:18.482430    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77 podName:18109ae3-f141-4b3a-aba4-37f3a8d82cdc nodeName:}" failed. No retries permitted until 2026-07-30 10:48:20.482411959 +0000 UTC m=+6.203296198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4l77" (UniqueName: "kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77") pod "network-check-target-x72dc" (UID: "18109ae3-f141-4b3a-aba4-37f3a8d82cdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:18.800012 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:18.799948    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 10:43:16 +0000 UTC" deadline="2028-03-03 00:42:09.172438428 +0000 UTC"
Jul 30 10:48:18.800012 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:18.799982    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13957h53m50.372460887s"
Jul 30 10:48:18.852161 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:18.851657    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:18.852161 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:18.851780    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:18.878059 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:18.878026    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal" event={"ID":"d15aaab8d00e7896dae80fe2b70d2f3b","Type":"ContainerStarted","Data":"cfd07e351c0ba218861a65505e73e6efcb7ba8b50d4544880c7de461cd39ba16"}
Jul 30 10:48:19.850542 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:19.849501    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:19.850542 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:19.849501    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:19.850542 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:19.849676    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:19.850542 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:19.849753    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:20.299036 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:20.298929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:20.299036 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:20.299019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:20.299291 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:20.299111    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:20.299291 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:20.299144    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:20.299291 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:20.299182    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret podName:20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d nodeName:}" failed. No retries permitted until 2026-07-30 10:48:24.299162752 +0000 UTC m=+10.020046994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret") pod "global-pull-secret-syncer-cbrck" (UID: "20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d") : object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:20.299291 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:20.299203    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs podName:c534fc6d-98cb-4a6e-b36a-1b8d6a61de79 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:24.29919258 +0000 UTC m=+10.020076823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs") pod "network-metrics-daemon-5qwx4" (UID: "c534fc6d-98cb-4a6e-b36a-1b8d6a61de79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:20.501175 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:20.501024    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l77\" (UniqueName: \"kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77\") pod \"network-check-target-x72dc\" (UID: \"18109ae3-f141-4b3a-aba4-37f3a8d82cdc\") " pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:20.501357 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:20.501332    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 10:48:20.501457 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:20.501389    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 10:48:20.501457 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:20.501403    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c4l77 for pod openshift-network-diagnostics/network-check-target-x72dc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:20.501549 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:20.501483    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77 podName:18109ae3-f141-4b3a-aba4-37f3a8d82cdc nodeName:}" failed. No retries permitted until 2026-07-30 10:48:24.5014623 +0000 UTC m=+10.222346542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4l77" (UniqueName: "kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77") pod "network-check-target-x72dc" (UID: "18109ae3-f141-4b3a-aba4-37f3a8d82cdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:20.851569 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:20.851542    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:20.852032 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:20.851661    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:21.849653 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:21.849180    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:21.849653 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:21.849321    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:21.849872 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:21.849657    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:21.849872 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:21.849780    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:22.849763 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:22.849723    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:22.850179 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:22.849868    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:23.850190 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:23.850152    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:23.850190 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:23.850176    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:23.850580 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:23.850293    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:23.850580 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:23.850433    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:24.334775 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:24.334682    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:24.334919 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:24.334776    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:24.334919 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:24.334854    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:24.335019 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:24.334856    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:24.335019 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:24.334963    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret podName:20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d nodeName:}" failed. No retries permitted until 2026-07-30 10:48:32.334943637 +0000 UTC m=+18.055827899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret") pod "global-pull-secret-syncer-cbrck" (UID: "20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d") : object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:24.335019 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:24.335016    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs podName:c534fc6d-98cb-4a6e-b36a-1b8d6a61de79 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:32.334991189 +0000 UTC m=+18.055875428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs") pod "network-metrics-daemon-5qwx4" (UID: "c534fc6d-98cb-4a6e-b36a-1b8d6a61de79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:24.537244 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:24.537206    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l77\" (UniqueName: \"kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77\") pod \"network-check-target-x72dc\" (UID: \"18109ae3-f141-4b3a-aba4-37f3a8d82cdc\") " pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:24.537454 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:24.537403    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 10:48:24.537454 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:24.537423    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 10:48:24.537454 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:24.537434    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c4l77 for pod openshift-network-diagnostics/network-check-target-x72dc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:24.537622 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:24.537502    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77 podName:18109ae3-f141-4b3a-aba4-37f3a8d82cdc nodeName:}" failed. No retries permitted until 2026-07-30 10:48:32.537483305 +0000 UTC m=+18.258367559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4l77" (UniqueName: "kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77") pod "network-check-target-x72dc" (UID: "18109ae3-f141-4b3a-aba4-37f3a8d82cdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:24.850347 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:24.850319    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:24.850826 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:24.850439    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:25.849956 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:25.849924    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:25.850150 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:25.850065    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:25.850150 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:25.850120    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:25.850281 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:25.850239    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:26.851857 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:26.851817    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:26.852435 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:26.851943    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:27.850074 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:27.850036    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:27.850074 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:27.850068    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:27.850331 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:27.850191    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:27.850392 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:27.850342    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:28.849776 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:28.849727    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:28.850272 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:28.849872    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:29.849159 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:29.849122    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:29.849345 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:29.849168    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:29.849345 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:29.849259    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:29.849448 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:29.849383    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:30.849638 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:30.849605    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:30.850055 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:30.849740    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:31.850140 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:31.850113    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:31.850588 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:31.850113    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:31.850588 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:31.850234    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:31.850588 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:31.850304    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:32.399449 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:32.399412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:32.399644 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:32.399471    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:32.399644 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:32.399617    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:32.399766 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:32.399688    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret podName:20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d nodeName:}" failed. No retries permitted until 2026-07-30 10:48:48.399668325 +0000 UTC m=+34.120552565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret") pod "global-pull-secret-syncer-cbrck" (UID: "20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d") : object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:32.399766 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:32.399617    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:32.399766 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:32.399763    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs podName:c534fc6d-98cb-4a6e-b36a-1b8d6a61de79 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:48.39974771 +0000 UTC m=+34.120631954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs") pod "network-metrics-daemon-5qwx4" (UID: "c534fc6d-98cb-4a6e-b36a-1b8d6a61de79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:32.601854 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:32.601821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l77\" (UniqueName: \"kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77\") pod \"network-check-target-x72dc\" (UID: \"18109ae3-f141-4b3a-aba4-37f3a8d82cdc\") " pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:32.602029 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:32.601961    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 10:48:32.602029 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:32.601982    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 10:48:32.602029 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:32.601992    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c4l77 for pod openshift-network-diagnostics/network-check-target-x72dc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:32.602196 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:32.602054    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77 podName:18109ae3-f141-4b3a-aba4-37f3a8d82cdc nodeName:}" failed. No retries permitted until 2026-07-30 10:48:48.602035392 +0000 UTC m=+34.322919643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4l77" (UniqueName: "kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77") pod "network-check-target-x72dc" (UID: "18109ae3-f141-4b3a-aba4-37f3a8d82cdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:32.853023 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:32.852944    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:32.853434 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:32.853072    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:33.850143 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:33.850093    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:33.850302 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:33.850154    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:33.850302 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:33.850234    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:33.850427 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:33.850329    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:34.850509 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:34.850478    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:34.850905 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:34.850586    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:35.850166 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.849824    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:35.850290 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:35.850203    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:35.850290 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.849859    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:35.850381 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:35.850319    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:35.910763 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.910736    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-grqcq" event={"ID":"90bb7436-3324-44f3-8e91-c4a972c6c38f","Type":"ContainerStarted","Data":"f1f01a7f33975e7e4adb042603b3d8a992bc78dc20f12843ac92e0d3da465bc7"}
Jul 30 10:48:35.912358 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.912338    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fl6ff" event={"ID":"444591cd-6538-4674-a92e-783563d56021","Type":"ContainerStarted","Data":"def6bb0e41f2dd3411ed445a5282bb55d6979dcac929cda60e32b3be07522eba"}
Jul 30 10:48:35.913651 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.913633    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4bfpq" event={"ID":"e881cdff-855a-4f17-9552-320d9b7a7daf","Type":"ContainerStarted","Data":"319bf9f780dd9bc3da1e50c0084c263631006e9c849e5e570d317202ce1feee5"}
Jul 30 10:48:35.914968 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.914949    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9t4zj" event={"ID":"d2a27b06-fcf7-4cbf-a646-ee312bada871","Type":"ContainerStarted","Data":"334cafcf8b6bf92ff47f1e71ea33b6089823f7fb62c94b114c3090d3bc750ae2"}
Jul 30 10:48:35.916395 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.916372    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf" event={"ID":"94ab27bc-722b-4c3e-84bb-97dff2669a57","Type":"ContainerStarted","Data":"7103549951eeb8a9b1ebbc27e1e27dafb23ae82865831e6701596e1380b5d50f"}
Jul 30 10:48:35.919093 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.919074    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr" event={"ID":"c8ccc70b-7123-458f-aeb5-d0f954af3515","Type":"ContainerStarted","Data":"81e7f3eae134091e27004bc7c7dc3d3847e45e6be29d591913e9b234c00076f8"}
Jul 30 10:48:35.919188 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.919156    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr" event={"ID":"c8ccc70b-7123-458f-aeb5-d0f954af3515","Type":"ContainerStarted","Data":"b949a76e53913354949d2b726c60468d4970076be94bf4d8ea886099f05d10a7"}
Jul 30 10:48:35.919235 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.919187    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr" event={"ID":"c8ccc70b-7123-458f-aeb5-d0f954af3515","Type":"ContainerStarted","Data":"c2cd0f9bc23d4496de411c6cefa915676be65a9a1f33f97bc7982824fdb02e0b"}
Jul 30 10:48:35.919235 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.919216    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr" event={"ID":"c8ccc70b-7123-458f-aeb5-d0f954af3515","Type":"ContainerStarted","Data":"a352dd945f71db5f25e2a1beba6403b71d0db7a2499eba8f07ee59f28ef14570"}
Jul 30 10:48:35.919323 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.919248    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr" event={"ID":"c8ccc70b-7123-458f-aeb5-d0f954af3515","Type":"ContainerStarted","Data":"8f7592fca09fcf3b907980e72d71ca2077a2b967fc2970c52b8b224dd22d7a35"}
Jul 30 10:48:35.919323 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.919289    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr" event={"ID":"c8ccc70b-7123-458f-aeb5-d0f954af3515","Type":"ContainerStarted","Data":"8c04eee5c20da54550248e567c06e9e2f5a7a0ee5e3cf3a8be7798fbe73046e6"}
Jul 30 10:48:35.920362 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.920342    2571 generic.go:356] "Generic (PLEG): container finished" podID="ab429725-9ff8-4ee4-97e8-e5c107e34235" containerID="539acc4bdfaa7d4b417bd7284d1f7f2a0257a97a0e1cdaa6fbeae37f3443b460" exitCode=0
Jul 30 10:48:35.920457 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.920402    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfr5" event={"ID":"ab429725-9ff8-4ee4-97e8-e5c107e34235","Type":"ContainerDied","Data":"539acc4bdfaa7d4b417bd7284d1f7f2a0257a97a0e1cdaa6fbeae37f3443b460"}
Jul 30 10:48:35.921822 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.921805    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wvnjw" event={"ID":"5df252dc-1dc8-40e4-a5f9-465fc2ee5819","Type":"ContainerStarted","Data":"83257c94d6a66204890d4853f0f591a741465a782fb64bd616a61c1ad7e2eb3c"}
Jul 30 10:48:35.928183 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.928148    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-grqcq" podStartSLOduration=13.3425892 podStartE2EDuration="21.92813632s" totalImagesPullingTime="8.58554712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:35.928136763 +0000 UTC m=+21.649021023" watchObservedRunningTime="2026-07-30 10:48:35.92813632 +0000 UTC m=+21.649020577"
Jul 30 10:48:35.928259 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.928241    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-243.ec2.internal" podStartSLOduration=20.928235831 podStartE2EDuration="20.928235831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:18.91735331 +0000 UTC m=+4.638237571" watchObservedRunningTime="2026-07-30 10:48:35.928235831 +0000 UTC m=+21.649120092"
Jul 30 10:48:35.943130 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.943075    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9t4zj" podStartSLOduration=8.221301847 podStartE2EDuration="21.943062919s" totalImagesPullingTime="13.721761072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:35.942717989 +0000 UTC m=+21.663602250" watchObservedRunningTime="2026-07-30 10:48:35.943062919 +0000 UTC m=+21.663947179"
Jul 30 10:48:35.962270 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:35.962238    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4bfpq" podStartSLOduration=4.321345505 podStartE2EDuration="21.962228423s" totalImagesPullingTime="17.640882918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:35.961859064 +0000 UTC m=+21.682743325" watchObservedRunningTime="2026-07-30 10:48:35.962228423 +0000 UTC m=+21.683112683"
Jul 30 10:48:36.001078 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:36.001045    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fl6ff" podStartSLOduration=4.304159181 podStartE2EDuration="22.001035585s" totalImagesPullingTime="17.696876404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:36.000865657 +0000 UTC m=+21.721749918" watchObservedRunningTime="2026-07-30 10:48:36.001035585 +0000 UTC m=+21.721919846"
Jul 30 10:48:36.017349 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:36.017319    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wvnjw" podStartSLOduration=4.378934595 podStartE2EDuration="22.017308592s" totalImagesPullingTime="17.638373997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:36.017204721 +0000 UTC m=+21.738088981" watchObservedRunningTime="2026-07-30 10:48:36.017308592 +0000 UTC m=+21.738192853"
Jul 30 10:48:36.855190 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:36.855161    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:36.855384 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:36.855283    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:36.926127 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:36.926074    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jrkr8" event={"ID":"9480f1c4-db3e-49b2-b4e3-a95469309c7b","Type":"ContainerStarted","Data":"c231c506ee41455c1c0b9087e7d7bf18d3b2d1ca557abdaa019daea3ee98298b"}
Jul 30 10:48:36.944270 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:36.944215    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jrkr8" podStartSLOduration=5.3025395379999996 podStartE2EDuration="22.944197203s" totalImagesPullingTime="17.641657665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:36.943421708 +0000 UTC m=+22.664305969" watchObservedRunningTime="2026-07-30 10:48:36.944197203 +0000 UTC m=+22.665081459"
Jul 30 10:48:37.849857 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:37.849835    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:37.849962 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:37.849933    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:37.850206 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:37.850180    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:37.850364 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:37.850343    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:37.933490 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:37.933453    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr" event={"ID":"c8ccc70b-7123-458f-aeb5-d0f954af3515","Type":"ContainerStarted","Data":"cadc527c86c924bb413c78aa7f0525f9a93f71507019c66f3f3ada46995f0f96"}
Jul 30 10:48:38.196019 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:38.195932    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:38.196537 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:38.196513    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:38.849694 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:38.849655    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:38.849865 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:38.849781    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:38.935528 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:38.935501    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:38.936080 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:38.936060    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9t4zj"
Jul 30 10:48:39.849924 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:39.849897    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:39.850116 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:39.849897    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:39.850116 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:39.850026    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:39.850116 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:39.850074    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:40.519736 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.519577    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 30 10:48:40.718504 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.718227    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T10:48:40.519730911Z","UUID":"96105eb8-4c7c-495d-acef-9e0bab3dae39","Handler":null,"Name":"","Endpoint":""}
Jul 30 10:48:40.719556 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.719537    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 30 10:48:40.719660 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.719571    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 30 10:48:40.849424 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.849396    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:40.849564 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:40.849490    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:40.940857 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.940824    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf" event={"ID":"94ab27bc-722b-4c3e-84bb-97dff2669a57","Type":"ContainerStarted","Data":"1f11097dac02ddf54b9828df8a1b628fa88228aadf839bd27d03a7243007563d"}
Jul 30 10:48:40.944075 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.944042    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr" event={"ID":"c8ccc70b-7123-458f-aeb5-d0f954af3515","Type":"ContainerStarted","Data":"0f901ca09c8984c2176c87beb21b6e1bc5bcb79a47efe0e95d94ec819c55b304"}
Jul 30 10:48:40.944330 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.944309    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:40.944423 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.944381    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:40.945932 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.945910    2571 generic.go:356] "Generic (PLEG): container finished" podID="ab429725-9ff8-4ee4-97e8-e5c107e34235" containerID="455eb6cfb822f5196fcef6bd5718cc2737a547e33f60e33fb43b147ccf2734a7" exitCode=0
Jul 30 10:48:40.946027 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.945998    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfr5" event={"ID":"ab429725-9ff8-4ee4-97e8-e5c107e34235","Type":"ContainerDied","Data":"455eb6cfb822f5196fcef6bd5718cc2737a547e33f60e33fb43b147ccf2734a7"}
Jul 30 10:48:40.960292 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:40.960273    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:41.051189 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:41.051045    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr" podStartSLOduration=9.316742093 podStartE2EDuration="27.051029177s" totalImagesPullingTime="17.734287084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:41.004838317 +0000 UTC m=+26.725722578" watchObservedRunningTime="2026-07-30 10:48:41.051029177 +0000 UTC m=+26.771913490"
Jul 30 10:48:41.849374 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:41.849345    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:41.849800 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:41.849494    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:41.849800 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:41.849580    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:41.849800 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:41.849660    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:41.950844 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:41.950798    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf" event={"ID":"94ab27bc-722b-4c3e-84bb-97dff2669a57","Type":"ContainerStarted","Data":"742577b6deb744b3193b44bd39f15485c0968453b2368506d2b5be34d578b8f7"}
Jul 30 10:48:41.951427 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:41.951407    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:41.965751 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:41.965728    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:48:41.973143 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:41.973088    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wsbkf" podStartSLOduration=4.162916356 podStartE2EDuration="27.973076252s" totalImagesPullingTime="23.810159896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:41.972674137 +0000 UTC m=+27.693558393" watchObservedRunningTime="2026-07-30 10:48:41.973076252 +0000 UTC m=+27.693960513"
Jul 30 10:48:42.122634 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:42.122536    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cbrck"]
Jul 30 10:48:42.122788 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:42.122656    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:42.122788 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:42.122758    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:42.127922 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:42.127898    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qwx4"]
Jul 30 10:48:42.128040 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:42.127986    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:42.128140 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:42.128086    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:42.128746 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:42.128714    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x72dc"]
Jul 30 10:48:42.128839 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:42.128822    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:42.128939 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:42.128919    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:42.955749 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:42.955531    2571 generic.go:356] "Generic (PLEG): container finished" podID="ab429725-9ff8-4ee4-97e8-e5c107e34235" containerID="428705dd0cada36bc068866178452f025ad241e8f050718dd73cc928d61a101b" exitCode=0
Jul 30 10:48:42.955749 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:42.955613    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfr5" event={"ID":"ab429725-9ff8-4ee4-97e8-e5c107e34235","Type":"ContainerDied","Data":"428705dd0cada36bc068866178452f025ad241e8f050718dd73cc928d61a101b"}
Jul 30 10:48:43.849660 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:43.849631    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:43.849820 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:43.849631    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:43.849820 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:43.849734    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:43.849890 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:43.849631    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:43.852608 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:43.850406    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:43.852789 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:43.852683    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:44.962500 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:44.962472    2571 generic.go:356] "Generic (PLEG): container finished" podID="ab429725-9ff8-4ee4-97e8-e5c107e34235" containerID="ea32260956603f2cbabadddfe22bbe374c5cccef86271cbddb83ba93d6df3b06" exitCode=0
Jul 30 10:48:44.963257 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:44.962503    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfr5" event={"ID":"ab429725-9ff8-4ee4-97e8-e5c107e34235","Type":"ContainerDied","Data":"ea32260956603f2cbabadddfe22bbe374c5cccef86271cbddb83ba93d6df3b06"}
Jul 30 10:48:45.849725 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:45.849687    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:45.849914 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:45.849816    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:45.849914 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:45.849824    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:45.849914 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:45.849852    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:45.850076 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:45.849956    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:45.850076 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:45.850057    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:47.849521 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:47.849484    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:47.849521 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:47.849505    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:47.850163 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:47.849513    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:47.850163 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:47.849615    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x72dc" podUID="18109ae3-f141-4b3a-aba4-37f3a8d82cdc"
Jul 30 10:48:47.850163 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:47.849722    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qwx4" podUID="c534fc6d-98cb-4a6e-b36a-1b8d6a61de79"
Jul 30 10:48:47.850163 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:47.849820    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbrck" podUID="20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d"
Jul 30 10:48:48.432625 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:48.432594    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:48.432827 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:48.432652    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:48.432827 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:48.432755    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:48.432827 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:48.432775    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:48.432827 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:48.432826    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs podName:c534fc6d-98cb-4a6e-b36a-1b8d6a61de79 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:20.43280855 +0000 UTC m=+66.153692789 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs") pod "network-metrics-daemon-5qwx4" (UID: "c534fc6d-98cb-4a6e-b36a-1b8d6a61de79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 10:48:48.433011 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:48.432846    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret podName:20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d nodeName:}" failed. No retries permitted until 2026-07-30 10:49:20.432836214 +0000 UTC m=+66.153720457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret") pod "global-pull-secret-syncer-cbrck" (UID: "20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d") : object "kube-system"/"original-pull-secret" not registered
Jul 30 10:48:48.634398 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:48.634355    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l77\" (UniqueName: \"kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77\") pod \"network-check-target-x72dc\" (UID: \"18109ae3-f141-4b3a-aba4-37f3a8d82cdc\") " pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:48.634596 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:48.634561    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 10:48:48.634596 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:48.634587    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 10:48:48.634718 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:48.634601    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c4l77 for pod openshift-network-diagnostics/network-check-target-x72dc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:48.634718 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:48.634663    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77 podName:18109ae3-f141-4b3a-aba4-37f3a8d82cdc nodeName:}" failed. No retries permitted until 2026-07-30 10:49:20.634644845 +0000 UTC m=+66.355529087 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4l77" (UniqueName: "kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77") pod "network-check-target-x72dc" (UID: "18109ae3-f141-4b3a-aba4-37f3a8d82cdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 10:48:49.106196 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.106176    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-243.ec2.internal" event="NodeReady"
Jul 30 10:48:49.106556 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.106320    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 30 10:48:49.140746 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.140726    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f5dd464d4-2qn6n"]
Jul 30 10:48:49.166980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.166789    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4swtg"]
Jul 30 10:48:49.167126 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.166994    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.170154 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.170124    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 30 10:48:49.170371 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.170174    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpkjw\""
Jul 30 10:48:49.170476 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.170396    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 30 10:48:49.170536 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.170491    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 30 10:48:49.178659 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.178639    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 30 10:48:49.182816 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.182796    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mzf92"]
Jul 30 10:48:49.182926 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.182902    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.185606 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.185590    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 30 10:48:49.185696 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.185591    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 30 10:48:49.185696 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.185594    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6l9vk\""
Jul 30 10:48:49.197932 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.197912    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f5dd464d4-2qn6n"]
Jul 30 10:48:49.197932 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.197912    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzf92"
Jul 30 10:48:49.198040 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.197970    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4swtg"]
Jul 30 10:48:49.198040 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.197989    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mzf92"]
Jul 30 10:48:49.201028 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.201010    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6r58j\""
Jul 30 10:48:49.201146 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.201045    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 30 10:48:49.201499 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.201485    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 30 10:48:49.201499 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.201492    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 30 10:48:49.339195 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339114    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aeeb651-2172-4d17-bcb5-4e78a6b84278-tmp-dir\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.339195 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339162    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwr6\" (UniqueName: \"kubernetes.io/projected/1bdbeb13-8dfd-42e8-97bc-f692b86db785-kube-api-access-xpwr6\") pod \"ingress-canary-mzf92\" (UID: \"1bdbeb13-8dfd-42e8-97bc-f692b86db785\") " pod="openshift-ingress-canary/ingress-canary-mzf92"
Jul 30 10:48:49.339195 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339192    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bdbeb13-8dfd-42e8-97bc-f692b86db785-cert\") pod \"ingress-canary-mzf92\" (UID: \"1bdbeb13-8dfd-42e8-97bc-f692b86db785\") " pod="openshift-ingress-canary/ingress-canary-mzf92"
Jul 30 10:48:49.339362 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339210    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk5f2\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-kube-api-access-sk5f2\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.339362 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339229    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.339362 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339252    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-registry-certificates\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.339362 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-bound-sa-token\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.339537 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339366    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeeb651-2172-4d17-bcb5-4e78a6b84278-config-volume\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.339537 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-trusted-ca\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.339537 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3152df11-49b1-4c5b-91be-024f914b72f5-ca-trust-extracted\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.339537 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.339717 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339557    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4fwf\" (UniqueName: \"kubernetes.io/projected/8aeeb651-2172-4d17-bcb5-4e78a6b84278-kube-api-access-b4fwf\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.339717 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-image-registry-private-configuration\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.339717 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.339625    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-installation-pull-secrets\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.440418 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440391    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeeb651-2172-4d17-bcb5-4e78a6b84278-config-volume\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.440520 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-trusted-ca\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.440520 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440453    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3152df11-49b1-4c5b-91be-024f914b72f5-ca-trust-extracted\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.440634 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440581    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.440690 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440659    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fwf\" (UniqueName: \"kubernetes.io/projected/8aeeb651-2172-4d17-bcb5-4e78a6b84278-kube-api-access-b4fwf\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.440690 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-image-registry-private-configuration\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.440797 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:49.440703    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 10:48:49.440797 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:49.440784    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls podName:8aeeb651-2172-4d17-bcb5-4e78a6b84278 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:49.940762907 +0000 UTC m=+35.661647166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls") pod "dns-default-4swtg" (UID: "8aeeb651-2172-4d17-bcb5-4e78a6b84278") : secret "dns-default-metrics-tls" not found
Jul 30 10:48:49.440903 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440707    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-installation-pull-secrets\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.440903 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440835    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3152df11-49b1-4c5b-91be-024f914b72f5-ca-trust-extracted\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.441053 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440896    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aeeb651-2172-4d17-bcb5-4e78a6b84278-tmp-dir\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.441053 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440936    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwr6\" (UniqueName: \"kubernetes.io/projected/1bdbeb13-8dfd-42e8-97bc-f692b86db785-kube-api-access-xpwr6\") pod \"ingress-canary-mzf92\" (UID: \"1bdbeb13-8dfd-42e8-97bc-f692b86db785\") " pod="openshift-ingress-canary/ingress-canary-mzf92"
Jul 30 10:48:49.441053 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440961    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bdbeb13-8dfd-42e8-97bc-f692b86db785-cert\") pod \"ingress-canary-mzf92\" (UID: \"1bdbeb13-8dfd-42e8-97bc-f692b86db785\") " pod="openshift-ingress-canary/ingress-canary-mzf92"
Jul 30 10:48:49.441053 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.440981    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk5f2\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-kube-api-access-sk5f2\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.441053 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.441020    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.441053 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.441040    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-registry-certificates\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.441418 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.441124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-bound-sa-token\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.441418 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.441215    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aeeb651-2172-4d17-bcb5-4e78a6b84278-tmp-dir\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.441418 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:49.441307    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 10:48:49.441418 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:49.441321    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5dd464d4-2qn6n: secret "image-registry-tls" not found
Jul 30 10:48:49.441418 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:49.441367    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls podName:3152df11-49b1-4c5b-91be-024f914b72f5 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:49.941350529 +0000 UTC m=+35.662234767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls") pod "image-registry-7f5dd464d4-2qn6n" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5") : secret "image-registry-tls" not found
Jul 30 10:48:49.441631 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.441042    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeeb651-2172-4d17-bcb5-4e78a6b84278-config-volume\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.441631 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.441536    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-trusted-ca\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.441807 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.441787    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-registry-certificates\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.444710 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.444684    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-installation-pull-secrets\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.444818 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.444729    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-image-registry-private-configuration\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.444818 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.444739    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bdbeb13-8dfd-42e8-97bc-f692b86db785-cert\") pod \"ingress-canary-mzf92\" (UID: \"1bdbeb13-8dfd-42e8-97bc-f692b86db785\") " pod="openshift-ingress-canary/ingress-canary-mzf92"
Jul 30 10:48:49.450570 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.450540    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk5f2\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-kube-api-access-sk5f2\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.450770 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.450751    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fwf\" (UniqueName: \"kubernetes.io/projected/8aeeb651-2172-4d17-bcb5-4e78a6b84278-kube-api-access-b4fwf\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.451148 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.451125    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwr6\" (UniqueName: \"kubernetes.io/projected/1bdbeb13-8dfd-42e8-97bc-f692b86db785-kube-api-access-xpwr6\") pod \"ingress-canary-mzf92\" (UID: \"1bdbeb13-8dfd-42e8-97bc-f692b86db785\") " pod="openshift-ingress-canary/ingress-canary-mzf92"
Jul 30 10:48:49.451861 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.451842    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-bound-sa-token\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.506817 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.506795    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzf92"
Jul 30 10:48:49.677084 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.677051    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mzf92"]
Jul 30 10:48:49.680405 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:48:49.680380    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdbeb13_8dfd_42e8_97bc_f692b86db785.slice/crio-5ea00e6ec0423e60c50c0ca0b393b6fa36529c69ff616c11ba0634eca0f0f2b8 WatchSource:0}: Error finding container 5ea00e6ec0423e60c50c0ca0b393b6fa36529c69ff616c11ba0634eca0f0f2b8: Status 404 returned error can't find the container with id 5ea00e6ec0423e60c50c0ca0b393b6fa36529c69ff616c11ba0634eca0f0f2b8
Jul 30 10:48:49.850168 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.850094    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:48:49.850281 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.850107    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:48:49.850504 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.850108    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:48:49.853731 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.853711    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 10:48:49.855286 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.855265    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwnnn\""
Jul 30 10:48:49.855398 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.855331    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 10:48:49.855556 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.855539    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gvflt\""
Jul 30 10:48:49.855649 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.855554    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 10:48:49.855649 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.855554    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 10:48:49.945016 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.944986    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:49.945160 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.945040    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:49.945213 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:49.945164    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 10:48:49.945213 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:49.945175    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 10:48:49.945213 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:49.945196    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5dd464d4-2qn6n: secret "image-registry-tls" not found
Jul 30 10:48:49.945300 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:49.945221    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls podName:8aeeb651-2172-4d17-bcb5-4e78a6b84278 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:50.945205965 +0000 UTC m=+36.666090203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls") pod "dns-default-4swtg" (UID: "8aeeb651-2172-4d17-bcb5-4e78a6b84278") : secret "dns-default-metrics-tls" not found
Jul 30 10:48:49.945300 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:49.945248    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls podName:3152df11-49b1-4c5b-91be-024f914b72f5 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:50.945231872 +0000 UTC m=+36.666116128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls") pod "image-registry-7f5dd464d4-2qn6n" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5") : secret "image-registry-tls" not found
Jul 30 10:48:49.976979 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.976948    2571 generic.go:356] "Generic (PLEG): container finished" podID="ab429725-9ff8-4ee4-97e8-e5c107e34235" containerID="05d3de095df79c69384ca28844ba2ce096b7ab8202547866e14099fa0da5545a" exitCode=0
Jul 30 10:48:49.977112 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.977038    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfr5" event={"ID":"ab429725-9ff8-4ee4-97e8-e5c107e34235","Type":"ContainerDied","Data":"05d3de095df79c69384ca28844ba2ce096b7ab8202547866e14099fa0da5545a"}
Jul 30 10:48:49.978329 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:49.978172    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mzf92" event={"ID":"1bdbeb13-8dfd-42e8-97bc-f692b86db785","Type":"ContainerStarted","Data":"5ea00e6ec0423e60c50c0ca0b393b6fa36529c69ff616c11ba0634eca0f0f2b8"}
Jul 30 10:48:50.955321 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:50.955062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:50.955764 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:50.955203    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 10:48:50.955764 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:50.955418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:50.955764 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:50.955435    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls podName:8aeeb651-2172-4d17-bcb5-4e78a6b84278 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:52.955419947 +0000 UTC m=+38.676304186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls") pod "dns-default-4swtg" (UID: "8aeeb651-2172-4d17-bcb5-4e78a6b84278") : secret "dns-default-metrics-tls" not found
Jul 30 10:48:50.955764 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:50.955524    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 10:48:50.955764 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:50.955536    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5dd464d4-2qn6n: secret "image-registry-tls" not found
Jul 30 10:48:50.955764 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:50.955580    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls podName:3152df11-49b1-4c5b-91be-024f914b72f5 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:52.955568992 +0000 UTC m=+38.676453234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls") pod "image-registry-7f5dd464d4-2qn6n" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5") : secret "image-registry-tls" not found
Jul 30 10:48:50.983689 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:50.983662    2571 generic.go:356] "Generic (PLEG): container finished" podID="ab429725-9ff8-4ee4-97e8-e5c107e34235" containerID="291aeee02a140928dd66a8116975e491ce32505c0b4fedbbb0ed75fe389cd612" exitCode=0
Jul 30 10:48:50.983809 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:50.983724    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfr5" event={"ID":"ab429725-9ff8-4ee4-97e8-e5c107e34235","Type":"ContainerDied","Data":"291aeee02a140928dd66a8116975e491ce32505c0b4fedbbb0ed75fe389cd612"}
Jul 30 10:48:51.442568 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:51.442486    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:51.534414 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:51.534378    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:51.860081 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:51.859983    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:51.927641 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:51.927609    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:51.990674 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:51.990639    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfr5" event={"ID":"ab429725-9ff8-4ee4-97e8-e5c107e34235","Type":"ContainerStarted","Data":"60363d093ab0708d33c55e21531d06722f9e22ac5c6f2068d02d943844bc6f27"}
Jul 30 10:48:52.018707 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:52.018664    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mqfr5" podStartSLOduration=18.688627643 podStartE2EDuration="38.01865007s" totalImagesPullingTime="19.330022427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:52.016433262 +0000 UTC m=+37.737317522" watchObservedRunningTime="2026-07-30 10:48:52.01865007 +0000 UTC m=+37.739534330"
Jul 30 10:48:52.025919 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:52.025897    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:52.207013 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:52.206982    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:52.968018 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:52.967986    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:52.968190 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:52.968048    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:52.968190 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:52.968150    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 10:48:52.968190 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:52.968161    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 10:48:52.968190 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:52.968168    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5dd464d4-2qn6n: secret "image-registry-tls" not found
Jul 30 10:48:52.968315 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:52.968216    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls podName:3152df11-49b1-4c5b-91be-024f914b72f5 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:56.968200629 +0000 UTC m=+42.689084901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls") pod "image-registry-7f5dd464d4-2qn6n" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5") : secret "image-registry-tls" not found
Jul 30 10:48:52.968315 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:52.968229    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls podName:8aeeb651-2172-4d17-bcb5-4e78a6b84278 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:56.96822386 +0000 UTC m=+42.689108099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls") pod "dns-default-4swtg" (UID: "8aeeb651-2172-4d17-bcb5-4e78a6b84278") : secret "dns-default-metrics-tls" not found
Jul 30 10:48:52.994757 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:52.994723    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mzf92" event={"ID":"1bdbeb13-8dfd-42e8-97bc-f692b86db785","Type":"ContainerStarted","Data":"eab34d4b8999061ea61120deca0fef1b2b3db84432dca8ff38b662550ea4404f"}
Jul 30 10:48:53.014228 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:53.014185    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mzf92" podStartSLOduration=1.225175061 podStartE2EDuration="4.014168102s" totalImagesPullingTime="2.788993041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:53.012432056 +0000 UTC m=+38.733316320" watchObservedRunningTime="2026-07-30 10:48:53.014168102 +0000 UTC m=+38.735052364"
Jul 30 10:48:54.931934 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:54.931896    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:54.990152 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:54.990116    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:55.473660 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:55.473608    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:56.993683 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:56.993638    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:48:56.994167 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:56.993705    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:48:56.994167 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:56.993798    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 10:48:56.994167 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:56.993810    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 10:48:56.994167 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:56.993818    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5dd464d4-2qn6n: secret "image-registry-tls" not found
Jul 30 10:48:56.994167 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:56.993906    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls podName:8aeeb651-2172-4d17-bcb5-4e78a6b84278 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:04.993883863 +0000 UTC m=+50.714768120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls") pod "dns-default-4swtg" (UID: "8aeeb651-2172-4d17-bcb5-4e78a6b84278") : secret "dns-default-metrics-tls" not found
Jul 30 10:48:56.994167 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:56.993945    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls podName:3152df11-49b1-4c5b-91be-024f914b72f5 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:04.993931725 +0000 UTC m=+50.714815964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls") pod "image-registry-7f5dd464d4-2qn6n" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5") : secret "image-registry-tls" not found
Jul 30 10:48:57.501590 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.501556    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-spcnq"]
Jul 30 10:48:57.527951 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.527923    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:57.531713 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.531692    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-spcnq"]
Jul 30 10:48:57.531821 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.531805    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.535294 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.535273    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 30 10:48:57.535401 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.535274    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 30 10:48:57.535601 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.535582    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 30 10:48:57.536642 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.536621    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 30 10:48:57.536953 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.536932    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tdvpn\""
Jul 30 10:48:57.586685 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.586654    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:57.599065 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.599036    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e588b42-d61a-4087-8a87-8b941a382ad2-crio-socket\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.599194 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.599093    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e588b42-d61a-4087-8a87-8b941a382ad2-data-volume\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.599194 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.599139    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.599194 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.599187    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.599331 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.599210    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftgl\" (UniqueName: \"kubernetes.io/projected/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-api-access-pftgl\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.699654 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.699617    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.699654 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.699653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftgl\" (UniqueName: \"kubernetes.io/projected/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-api-access-pftgl\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.699889 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.699732    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e588b42-d61a-4087-8a87-8b941a382ad2-crio-socket\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.699889 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:57.699743    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 10:48:57.699889 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.699790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e588b42-d61a-4087-8a87-8b941a382ad2-data-volume\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.699889 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.699815    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.699889 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:57.699831    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm podName:5e588b42-d61a-4087-8a87-8b941a382ad2 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:58.199809856 +0000 UTC m=+43.920694117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-spcnq" (UID: "5e588b42-d61a-4087-8a87-8b941a382ad2") : configmap "kube-rbac-proxy" not found
Jul 30 10:48:57.700154 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:57.699925    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 10:48:57.700154 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.699965    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e588b42-d61a-4087-8a87-8b941a382ad2-crio-socket\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.700154 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:57.699982    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls podName:5e588b42-d61a-4087-8a87-8b941a382ad2 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:58.199965829 +0000 UTC m=+43.920850071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spcnq" (UID: "5e588b42-d61a-4087-8a87-8b941a382ad2") : secret "insights-runtime-extractor-tls" not found
Jul 30 10:48:57.700257 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.700186    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e588b42-d61a-4087-8a87-8b941a382ad2-data-volume\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.710194 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.710172    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftgl\" (UniqueName: \"kubernetes.io/projected/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-api-access-pftgl\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:57.764786 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.764714    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:57.864991 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:57.864955    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:58.203339 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:58.203305    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:58.203736 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:58.203355    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:58.203736 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:58.203459    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 10:48:58.203736 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:58.203485    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 10:48:58.203736 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:58.203522    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls podName:5e588b42-d61a-4087-8a87-8b941a382ad2 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:59.2035065 +0000 UTC m=+44.924390764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spcnq" (UID: "5e588b42-d61a-4087-8a87-8b941a382ad2") : secret "insights-runtime-extractor-tls" not found
Jul 30 10:48:58.204689 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:58.204662    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm podName:5e588b42-d61a-4087-8a87-8b941a382ad2 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:59.204637126 +0000 UTC m=+44.925521382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-spcnq" (UID: "5e588b42-d61a-4087-8a87-8b941a382ad2") : configmap "kube-rbac-proxy" not found
Jul 30 10:48:58.235498 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:58.235470    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:58.327071 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:58.327040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:58.428208 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:58.428173    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:48:59.211188 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.211144    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:59.211567 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.211203    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:48:59.211567 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:59.211270    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 10:48:59.211567 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:59.211310    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 10:48:59.211567 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:59.211321    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm podName:5e588b42-d61a-4087-8a87-8b941a382ad2 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:01.211308278 +0000 UTC m=+46.932192516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-spcnq" (UID: "5e588b42-d61a-4087-8a87-8b941a382ad2") : configmap "kube-rbac-proxy" not found
Jul 30 10:48:59.211567 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:48:59.211385    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls podName:5e588b42-d61a-4087-8a87-8b941a382ad2 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:01.211365182 +0000 UTC m=+46.932249424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spcnq" (UID: "5e588b42-d61a-4087-8a87-8b941a382ad2") : secret "insights-runtime-extractor-tls" not found
Jul 30 10:48:59.495176 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.495087    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-grqcq_90bb7436-3324-44f3-8e91-c4a972c6c38f/dns-node-resolver/0.log"
Jul 30 10:48:59.825443 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.825365    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-bglqf"]
Jul 30 10:48:59.844522 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.844497    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-bglqf"]
Jul 30 10:48:59.844654 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.844611    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:48:59.847743 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.847719    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 30 10:48:59.847873 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.847835    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 30 10:48:59.847935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.847919    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 30 10:48:59.849180 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.849163    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 30 10:48:59.849362 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.849347    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zwxcs\""
Jul 30 10:48:59.916667 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.916631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/54342377-963b-46f5-aabb-4ee4471fca01-signing-key\") pod \"service-ca-658d7b7c78-bglqf\" (UID: \"54342377-963b-46f5-aabb-4ee4471fca01\") " pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:48:59.916842 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.916721    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/54342377-963b-46f5-aabb-4ee4471fca01-signing-cabundle\") pod \"service-ca-658d7b7c78-bglqf\" (UID: \"54342377-963b-46f5-aabb-4ee4471fca01\") " pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:48:59.916842 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.916740    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5cxf\" (UniqueName: \"kubernetes.io/projected/54342377-963b-46f5-aabb-4ee4471fca01-kube-api-access-t5cxf\") pod \"service-ca-658d7b7c78-bglqf\" (UID: \"54342377-963b-46f5-aabb-4ee4471fca01\") " pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:48:59.937730 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:48:59.937699    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:00.017921 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:00.017890    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/54342377-963b-46f5-aabb-4ee4471fca01-signing-cabundle\") pod \"service-ca-658d7b7c78-bglqf\" (UID: \"54342377-963b-46f5-aabb-4ee4471fca01\") " pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:49:00.018075 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:00.017927    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5cxf\" (UniqueName: \"kubernetes.io/projected/54342377-963b-46f5-aabb-4ee4471fca01-kube-api-access-t5cxf\") pod \"service-ca-658d7b7c78-bglqf\" (UID: \"54342377-963b-46f5-aabb-4ee4471fca01\") " pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:49:00.018075 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:00.017975    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/54342377-963b-46f5-aabb-4ee4471fca01-signing-key\") pod \"service-ca-658d7b7c78-bglqf\" (UID: \"54342377-963b-46f5-aabb-4ee4471fca01\") " pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:49:00.018622 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:00.018596    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/54342377-963b-46f5-aabb-4ee4471fca01-signing-cabundle\") pod \"service-ca-658d7b7c78-bglqf\" (UID: \"54342377-963b-46f5-aabb-4ee4471fca01\") " pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:49:00.020413 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:00.020396    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/54342377-963b-46f5-aabb-4ee4471fca01-signing-key\") pod \"service-ca-658d7b7c78-bglqf\" (UID: \"54342377-963b-46f5-aabb-4ee4471fca01\") " pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:49:00.027122 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:00.027087    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5cxf\" (UniqueName: \"kubernetes.io/projected/54342377-963b-46f5-aabb-4ee4471fca01-kube-api-access-t5cxf\") pod \"service-ca-658d7b7c78-bglqf\" (UID: \"54342377-963b-46f5-aabb-4ee4471fca01\") " pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:49:00.153571 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:00.153483    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-bglqf"
Jul 30 10:49:00.288994 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:00.288962    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-bglqf"]
Jul 30 10:49:00.292712 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:49:00.292685    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54342377_963b_46f5_aabb_4ee4471fca01.slice/crio-63314963f16141ffc0622d74d88c5a9a9cbe7e5852cd06bdc536bad492814413 WatchSource:0}: Error finding container 63314963f16141ffc0622d74d88c5a9a9cbe7e5852cd06bdc536bad492814413: Status 404 returned error can't find the container with id 63314963f16141ffc0622d74d88c5a9a9cbe7e5852cd06bdc536bad492814413
Jul 30 10:49:00.694832 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:00.694805    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wvnjw_5df252dc-1dc8-40e4-a5f9-465fc2ee5819/node-ca/0.log"
Jul 30 10:49:00.723286 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:00.723255    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:01.014660 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:01.014579    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-bglqf" event={"ID":"54342377-963b-46f5-aabb-4ee4471fca01","Type":"ContainerStarted","Data":"63314963f16141ffc0622d74d88c5a9a9cbe7e5852cd06bdc536bad492814413"}
Jul 30 10:49:01.225785 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:01.225749    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:49:01.225953 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:01.225804    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:49:01.225953 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:01.225864    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 10:49:01.225953 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:01.225911    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm podName:5e588b42-d61a-4087-8a87-8b941a382ad2 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:05.225898666 +0000 UTC m=+50.946782906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-spcnq" (UID: "5e588b42-d61a-4087-8a87-8b941a382ad2") : configmap "kube-rbac-proxy" not found
Jul 30 10:49:01.225953 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:01.225930    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 10:49:01.226089 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:01.225998    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls podName:5e588b42-d61a-4087-8a87-8b941a382ad2 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:05.225984672 +0000 UTC m=+50.946868911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spcnq" (UID: "5e588b42-d61a-4087-8a87-8b941a382ad2") : secret "insights-runtime-extractor-tls" not found
Jul 30 10:49:01.494916 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:01.494886    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mzf92_1bdbeb13-8dfd-42e8-97bc-f692b86db785/serve-healthcheck-canary/0.log"
Jul 30 10:49:02.110980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:02.110945    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:02.169817 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:02.169783    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:03.021610 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:03.021558    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-bglqf" event={"ID":"54342377-963b-46f5-aabb-4ee4471fca01","Type":"ContainerStarted","Data":"00ea92c6d8540a7643af587d106a9496a48959c1667a979dc4938eee59f3472b"}
Jul 30 10:49:03.046078 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:03.046028    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-bglqf" podStartSLOduration=1.483264058 podStartE2EDuration="4.046012242s" totalImagesPullingTime="2.562748184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:03.045086495 +0000 UTC m=+48.765970758" watchObservedRunningTime="2026-07-30 10:49:03.046012242 +0000 UTC m=+48.766896502"
Jul 30 10:49:03.336193 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:03.336142    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:03.962446 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:03.962416    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:04.068026 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:04.067987    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:04.168591 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:04.168547    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:04.268907 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:04.268800    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:04.331895 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:04.331850    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:04.657851 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:04.657771    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:05.060053 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:05.060024    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:49:05.060246 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:05.060081    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:49:05.060246 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:05.060173    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 10:49:05.060246 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:05.060191    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5dd464d4-2qn6n: secret "image-registry-tls" not found
Jul 30 10:49:05.060246 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:05.060223    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 10:49:05.060246 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:05.060241    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls podName:3152df11-49b1-4c5b-91be-024f914b72f5 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:21.06022548 +0000 UTC m=+66.781109722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls") pod "image-registry-7f5dd464d4-2qn6n" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5") : secret "image-registry-tls" not found
Jul 30 10:49:05.060429 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:05.060262    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls podName:8aeeb651-2172-4d17-bcb5-4e78a6b84278 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:21.060250936 +0000 UTC m=+66.781135216 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls") pod "dns-default-4swtg" (UID: "8aeeb651-2172-4d17-bcb5-4e78a6b84278") : secret "dns-default-metrics-tls" not found
Jul 30 10:49:05.261581 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:05.261547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:49:05.262027 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:05.261593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:49:05.262027 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:05.261722    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 10:49:05.262027 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:05.261804    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls podName:5e588b42-d61a-4087-8a87-8b941a382ad2 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:13.261783544 +0000 UTC m=+58.982667786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-spcnq" (UID: "5e588b42-d61a-4087-8a87-8b941a382ad2") : secret "insights-runtime-extractor-tls" not found
Jul 30 10:49:05.262227 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:05.262072    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e588b42-d61a-4087-8a87-8b941a382ad2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:49:05.657308 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:05.657276    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:05.858823 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:05.858787    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:06.262566 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:06.262531    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:06.455333 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:06.455297    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:06.864041 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:06.864007    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:07.508738 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:07.508704    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:07.610143 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:07.610111    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:07.709716 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:07.709684    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:07.864480 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:07.864392    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:08.179635 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:08.179602    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:08.280912 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:08.280881    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:08.654744 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:08.654669    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:08.856224 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:08.856176    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:08.917916 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:08.917820    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:09.460040 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:09.459995    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:09.560267 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:09.560236    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:09.659195 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:09.659166    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:10.058749 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:10.058713    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:10.256690 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:10.256631    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:10.377493 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:10.377370    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:10.658625 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:10.658551    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:10.855065 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:10.855037    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:11.261183 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:11.261150    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:12.146713 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:12.146681    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:12.460874 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:12.460844    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:12.659161 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:12.659125    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:12.857681 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:12.857613    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:13.262040 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:13.262004    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:13.327340 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:13.327312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:49:13.329582 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:13.329555    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e588b42-d61a-4087-8a87-8b941a382ad2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spcnq\" (UID: \"5e588b42-d61a-4087-8a87-8b941a382ad2\") " pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:49:13.440560 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:13.440531    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-spcnq"
Jul 30 10:49:13.603013 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:13.602980    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-spcnq"]
Jul 30 10:49:13.857342 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:13.857270    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:13.972765 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:13.972740    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qltbr"
Jul 30 10:49:14.048084 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:14.048051    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spcnq" event={"ID":"5e588b42-d61a-4087-8a87-8b941a382ad2","Type":"ContainerStarted","Data":"1e6c078eae4990d9cf0e52514bad5e257d7ea36c45b9cab41e800a2e671fcb58"}
Jul 30 10:49:14.257090 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:14.257056    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:14.461754 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:14.461715    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:14.656444 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:14.656332    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:17.112079 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:17.111974    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:17.212110 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:17.212050    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:19.063619 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:19.063536    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spcnq" event={"ID":"5e588b42-d61a-4087-8a87-8b941a382ad2","Type":"ContainerStarted","Data":"8428ae5a5d7d24dbb771bab125b84ae1783d040b507d5d391a7edba2e93feeb1"}
Jul 30 10:49:20.489704 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.489661    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:49:20.490181 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.489767    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:49:20.493006 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.492984    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 10:49:20.494080 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.494049    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 10:49:20.502712 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.502692    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d-original-pull-secret\") pod \"global-pull-secret-syncer-cbrck\" (UID: \"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d\") " pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:49:20.502841 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.502820    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c534fc6d-98cb-4a6e-b36a-1b8d6a61de79-metrics-certs\") pod \"network-metrics-daemon-5qwx4\" (UID: \"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79\") " pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:49:20.690827 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.690781    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l77\" (UniqueName: \"kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77\") pod \"network-check-target-x72dc\" (UID: \"18109ae3-f141-4b3a-aba4-37f3a8d82cdc\") " pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:49:20.693725 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.693695    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 10:49:20.704572 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.704542    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 10:49:20.715368 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.715313    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4l77\" (UniqueName: \"kubernetes.io/projected/18109ae3-f141-4b3a-aba4-37f3a8d82cdc-kube-api-access-c4l77\") pod \"network-check-target-x72dc\" (UID: \"18109ae3-f141-4b3a-aba4-37f3a8d82cdc\") " pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:49:20.763375 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.763301    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gvflt\""
Jul 30 10:49:20.769170 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.769146    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwnnn\""
Jul 30 10:49:20.770207 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.770184    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbrck"
Jul 30 10:49:20.771262 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.771241    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qwx4"
Jul 30 10:49:20.777488 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.777463    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:49:20.860140 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.860112    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f5dd464d4-2qn6n"]
Jul 30 10:49:20.860379 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:49:20.860350    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n" podUID="3152df11-49b1-4c5b-91be-024f914b72f5"
Jul 30 10:49:20.880788 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.880755    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"]
Jul 30 10:49:20.916045 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.916018    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"]
Jul 30 10:49:20.916206 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.916169    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:20.918896 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.918841    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:20.919187 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.919162    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 30 10:49:20.919359 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.919162    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 30 10:49:20.919796 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.919579    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 30 10:49:20.919796 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.919603    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 30 10:49:20.939988 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.939965    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"]
Jul 30 10:49:20.940135 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.940014    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"]
Jul 30 10:49:20.940135 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.940112    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:20.945053 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.945028    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 30 10:49:20.945180 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.945088    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 30 10:49:20.945728 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.945710    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 30 10:49:20.948191 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.948166    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 30 10:49:20.993182 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.993142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4lp\" (UniqueName: \"kubernetes.io/projected/661a2c0b-95c5-4e58-ac6e-e9a7ca12f430-kube-api-access-vx4lp\") pod \"klusterlet-addon-workmgr-644b68b5f-52nxl\" (UID: \"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:20.993334 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.993203    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/661a2c0b-95c5-4e58-ac6e-e9a7ca12f430-tmp\") pod \"klusterlet-addon-workmgr-644b68b5f-52nxl\" (UID: \"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:20.993334 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:20.993231    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/661a2c0b-95c5-4e58-ac6e-e9a7ca12f430-klusterlet-config\") pod \"klusterlet-addon-workmgr-644b68b5f-52nxl\" (UID: \"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:21.018122 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.018020    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:21.068627 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.068590    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:49:21.094084 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:49:21.094270 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094092    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4lp\" (UniqueName: \"kubernetes.io/projected/661a2c0b-95c5-4e58-ac6e-e9a7ca12f430-kube-api-access-vx4lp\") pod \"klusterlet-addon-workmgr-644b68b5f-52nxl\" (UID: \"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:21.094270 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094149    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-hub\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.094270 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094179    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-ca\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.094270 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094221    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.094482 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094290    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz57c\" (UniqueName: \"kubernetes.io/projected/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-kube-api-access-gz57c\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.094482 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094380    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/661a2c0b-95c5-4e58-ac6e-e9a7ca12f430-klusterlet-config\") pod \"klusterlet-addon-workmgr-644b68b5f-52nxl\" (UID: \"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:21.094482 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094420    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:49:21.094482 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094448    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.094482 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094481    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/661a2c0b-95c5-4e58-ac6e-e9a7ca12f430-tmp\") pod \"klusterlet-addon-workmgr-644b68b5f-52nxl\" (UID: \"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:21.094717 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094505    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.094995 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.094936    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/661a2c0b-95c5-4e58-ac6e-e9a7ca12f430-tmp\") pod \"klusterlet-addon-workmgr-644b68b5f-52nxl\" (UID: \"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:21.097093 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.097070    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aeeb651-2172-4d17-bcb5-4e78a6b84278-metrics-tls\") pod \"dns-default-4swtg\" (UID: \"8aeeb651-2172-4d17-bcb5-4e78a6b84278\") " pod="openshift-dns/dns-default-4swtg"
Jul 30 10:49:21.097202 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.097175    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") pod \"image-registry-7f5dd464d4-2qn6n\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") " pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:49:21.097981 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.097938    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/661a2c0b-95c5-4e58-ac6e-e9a7ca12f430-klusterlet-config\") pod \"klusterlet-addon-workmgr-644b68b5f-52nxl\" (UID: \"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:21.103222 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.103198    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4lp\" (UniqueName: \"kubernetes.io/projected/661a2c0b-95c5-4e58-ac6e-e9a7ca12f430-kube-api-access-vx4lp\") pod \"klusterlet-addon-workmgr-644b68b5f-52nxl\" (UID: \"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:21.106862 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.106816    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:21.128550 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.128514    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:49:21.194947 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.194918    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-trusted-ca\") pod \"3152df11-49b1-4c5b-91be-024f914b72f5\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") "
Jul 30 10:49:21.195135 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.194963    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk5f2\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-kube-api-access-sk5f2\") pod \"3152df11-49b1-4c5b-91be-024f914b72f5\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") "
Jul 30 10:49:21.195135 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.194997    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3152df11-49b1-4c5b-91be-024f914b72f5-ca-trust-extracted\") pod \"3152df11-49b1-4c5b-91be-024f914b72f5\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") "
Jul 30 10:49:21.195135 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195015    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-installation-pull-secrets\") pod \"3152df11-49b1-4c5b-91be-024f914b72f5\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") "
Jul 30 10:49:21.195312 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195187    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-image-registry-private-configuration\") pod \"3152df11-49b1-4c5b-91be-024f914b72f5\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") "
Jul 30 10:49:21.195312 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195238    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-registry-certificates\") pod \"3152df11-49b1-4c5b-91be-024f914b72f5\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") "
Jul 30 10:49:21.195312 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195270    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") pod \"3152df11-49b1-4c5b-91be-024f914b72f5\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") "
Jul 30 10:49:21.195312 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195298    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-bound-sa-token\") pod \"3152df11-49b1-4c5b-91be-024f914b72f5\" (UID: \"3152df11-49b1-4c5b-91be-024f914b72f5\") "
Jul 30 10:49:21.195496 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195392    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.195496 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195416    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3152df11-49b1-4c5b-91be-024f914b72f5" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:49:21.195496 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195441    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.195644 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195590    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-hub\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.195644 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195605    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3152df11-49b1-4c5b-91be-024f914b72f5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3152df11-49b1-4c5b-91be-024f914b72f5" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 10:49:21.195644 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195635    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-ca\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.195795 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.195795 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz57c\" (UniqueName: \"kubernetes.io/projected/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-kube-api-access-gz57c\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.195897 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195868    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3152df11-49b1-4c5b-91be-024f914b72f5-ca-trust-extracted\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:49:21.195897 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.195886    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-trusted-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:49:21.197056 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.197029    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3152df11-49b1-4c5b-91be-024f914b72f5" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:49:21.197348 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.197322    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.198434 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.198404    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.198553 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.198480    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-ca\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.199070 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.199046    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.199528 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.199495    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3152df11-49b1-4c5b-91be-024f914b72f5" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 10:49:21.199528 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.199503    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-kube-api-access-sk5f2" (OuterVolumeSpecName: "kube-api-access-sk5f2") pod "3152df11-49b1-4c5b-91be-024f914b72f5" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5"). InnerVolumeSpecName "kube-api-access-sk5f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 10:49:21.199528 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.199516    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3152df11-49b1-4c5b-91be-024f914b72f5" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 10:49:21.199672 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.199591    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3152df11-49b1-4c5b-91be-024f914b72f5" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 10:49:21.199672 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.199639    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3152df11-49b1-4c5b-91be-024f914b72f5" (UID: "3152df11-49b1-4c5b-91be-024f914b72f5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 10:49:21.200548 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.200526    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-hub\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.207437 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.207416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz57c\" (UniqueName: \"kubernetes.io/projected/400e5d0e-2ec4-4ecd-b02a-af44fd3079f6-kube-api-access-gz57c\") pod \"cluster-proxy-proxy-agent-66d5b6689f-lzb94\" (UID: \"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.208500 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.208480    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:21.226842 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.226819    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:21.261182 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.261150    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"
Jul 30 10:49:21.296175 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.296152    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-installation-pull-secrets\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:49:21.296175 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.296175    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3152df11-49b1-4c5b-91be-024f914b72f5-image-registry-private-configuration\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:49:21.296301 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.296184    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3152df11-49b1-4c5b-91be-024f914b72f5-registry-certificates\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:49:21.296301 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.296195    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-registry-tls\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:49:21.296301 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.296204    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-bound-sa-token\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:49:21.296301 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.296211    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk5f2\" (UniqueName: \"kubernetes.io/projected/3152df11-49b1-4c5b-91be-024f914b72f5-kube-api-access-sk5f2\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:49:21.298727 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.298710    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6l9vk\""
Jul 30 10:49:21.306511 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:21.306490    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4swtg"
Jul 30 10:49:22.071299 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:22.071266    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5dd464d4-2qn6n"
Jul 30 10:49:22.108614 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:22.108584    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f5dd464d4-2qn6n"]
Jul 30 10:49:22.114228 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:22.114206    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f5dd464d4-2qn6n"]
Jul 30 10:49:22.445462 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:22.445389    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x72dc"]
Jul 30 10:49:22.464935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:22.464907    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qwx4"]
Jul 30 10:49:22.607237 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:49:22.607147    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18109ae3_f141_4b3a_aba4_37f3a8d82cdc.slice/crio-4bf108025b2f0ccc70578b6694794bf105f2fbb4fcbf43daf48f525d18cff05c WatchSource:0}: Error finding container 4bf108025b2f0ccc70578b6694794bf105f2fbb4fcbf43daf48f525d18cff05c: Status 404 returned error can't find the container with id 4bf108025b2f0ccc70578b6694794bf105f2fbb4fcbf43daf48f525d18cff05c
Jul 30 10:49:22.607476 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:49:22.607439    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc534fc6d_98cb_4a6e_b36a_1b8d6a61de79.slice/crio-4d6a6c6289af0ff35a43ed3a39d97f758f1febc50a0948e4ea027ac4f35c3a4c WatchSource:0}: Error finding container 4d6a6c6289af0ff35a43ed3a39d97f758f1febc50a0948e4ea027ac4f35c3a4c: Status 404 returned error can't find the container with id 4d6a6c6289af0ff35a43ed3a39d97f758f1febc50a0948e4ea027ac4f35c3a4c
Jul 30 10:49:22.689732 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:22.689707    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"]
Jul 30 10:49:22.695163 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:22.695092    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94"]
Jul 30 10:49:22.700433 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:49:22.700410    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661a2c0b_95c5_4e58_ac6e_e9a7ca12f430.slice/crio-09d08d2b042a4e2e0ddd604dc47ae0b3e44f578af4268cbd868d577629c0ade4 WatchSource:0}: Error finding container 09d08d2b042a4e2e0ddd604dc47ae0b3e44f578af4268cbd868d577629c0ade4: Status 404 returned error can't find the container with id 09d08d2b042a4e2e0ddd604dc47ae0b3e44f578af4268cbd868d577629c0ade4
Jul 30 10:49:22.700704 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:49:22.700684    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod400e5d0e_2ec4_4ecd_b02a_af44fd3079f6.slice/crio-d856780e772e9ce771f186c4af140337907cd75cf81656fd9b41428556ff215e WatchSource:0}: Error finding container d856780e772e9ce771f186c4af140337907cd75cf81656fd9b41428556ff215e: Status 404 returned error can't find the container with id d856780e772e9ce771f186c4af140337907cd75cf81656fd9b41428556ff215e
Jul 30 10:49:22.721730 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:22.721707    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cbrck"]
Jul 30 10:49:22.725897 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:22.724422    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4swtg"]
Jul 30 10:49:22.725897 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:49:22.725093    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d1cebf_7571_4c3b_b4fa_c6f6014a7e2d.slice/crio-543db6288378e7ac830129deeb5e0d667b5867fcc572fcd4b61b92b72d42de91 WatchSource:0}: Error finding container 543db6288378e7ac830129deeb5e0d667b5867fcc572fcd4b61b92b72d42de91: Status 404 returned error can't find the container with id 543db6288378e7ac830129deeb5e0d667b5867fcc572fcd4b61b92b72d42de91
Jul 30 10:49:22.728821 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:49:22.728799    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aeeb651_2172_4d17_bcb5_4e78a6b84278.slice/crio-646387c790f2fce076ffc85f8b5ef2329f7c7ac9865b33ff5eadb01ff892d8f3 WatchSource:0}: Error finding container 646387c790f2fce076ffc85f8b5ef2329f7c7ac9865b33ff5eadb01ff892d8f3: Status 404 returned error can't find the container with id 646387c790f2fce076ffc85f8b5ef2329f7c7ac9865b33ff5eadb01ff892d8f3
Jul 30 10:49:22.852666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:22.852632    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3152df11-49b1-4c5b-91be-024f914b72f5" path="/var/lib/kubelet/pods/3152df11-49b1-4c5b-91be-024f914b72f5/volumes"
Jul 30 10:49:23.076854 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:23.076821    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spcnq" event={"ID":"5e588b42-d61a-4087-8a87-8b941a382ad2","Type":"ContainerStarted","Data":"e4610ba4d2cf65a39a26d6f2827ecb123aa80ee81ed0ce73967bee81252111a4"}
Jul 30 10:49:23.078139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:23.078065    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94" event={"ID":"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6","Type":"ContainerStarted","Data":"d856780e772e9ce771f186c4af140337907cd75cf81656fd9b41428556ff215e"}
Jul 30 10:49:23.079381 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:23.079346    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qwx4" event={"ID":"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79","Type":"ContainerStarted","Data":"4d6a6c6289af0ff35a43ed3a39d97f758f1febc50a0948e4ea027ac4f35c3a4c"}
Jul 30 10:49:23.080464 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:23.080439    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4swtg" event={"ID":"8aeeb651-2172-4d17-bcb5-4e78a6b84278","Type":"ContainerStarted","Data":"646387c790f2fce076ffc85f8b5ef2329f7c7ac9865b33ff5eadb01ff892d8f3"}
Jul 30 10:49:23.081930 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:23.081905    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl" event={"ID":"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430","Type":"ContainerStarted","Data":"09d08d2b042a4e2e0ddd604dc47ae0b3e44f578af4268cbd868d577629c0ade4"}
Jul 30 10:49:23.083272 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:23.083244    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cbrck" event={"ID":"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d","Type":"ContainerStarted","Data":"543db6288378e7ac830129deeb5e0d667b5867fcc572fcd4b61b92b72d42de91"}
Jul 30 10:49:23.084566 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:23.084524    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x72dc" event={"ID":"18109ae3-f141-4b3a-aba4-37f3a8d82cdc","Type":"ContainerStarted","Data":"4bf108025b2f0ccc70578b6694794bf105f2fbb4fcbf43daf48f525d18cff05c"}
Jul 30 10:49:23.280086 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:23.279930    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:23.669529 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:23.669475    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:23.723800 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:23.723756    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:24.963775 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:24.963740    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:25.066998 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:25.066966    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:25.107038 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:25.106993    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qwx4" event={"ID":"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79","Type":"ContainerStarted","Data":"5febf1023ce7352cdb9835c030b05ef73288d5b01aeb940fd3e6f45ed7d0ed7c"}
Jul 30 10:49:25.107206 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:25.107061    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qwx4" event={"ID":"c534fc6d-98cb-4a6e-b36a-1b8d6a61de79","Type":"ContainerStarted","Data":"47fcf6cf9b7281d8674f15785437b3bcde1bb8b3525478c0dcf9d4aaac5153b2"}
Jul 30 10:49:25.133939 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:25.133883    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:26.077040 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:26.076870    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:26.462607 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:26.462575    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:26.519314 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:26.519278    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:27.493834 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:27.493779    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:27.594702 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:27.594661    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:27.671920 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:27.671867    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:27.771662 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:27.771578    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:29.546306 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:29.546269    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:29.641399 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:29.641362    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:29.808347 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:29.808264    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:29.908233 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:29.908191    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:30.009379 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:30.009342    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:30.075263 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:30.075182    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:30.441482 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:30.441442    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:30.564523 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:30.564490    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:30.662786 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:30.662750    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:30.832360 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:30.832267    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:30.896983 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:30.896948    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:31.224074 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:31.224038    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:31.423787 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:31.423753    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:31.862355 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:31.862320    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:32.009552 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:32.009513    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:32.111254 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:32.111214    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:32.171963 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:32.171840    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:32.439541 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:32.439513    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:32.540965 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:32.540935    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:33.694962 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:33.694931    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:33.803751 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:33.803717    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:34.099784 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:34.099699    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:35.147398 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.147355    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x72dc" event={"ID":"18109ae3-f141-4b3a-aba4-37f3a8d82cdc","Type":"ContainerStarted","Data":"b491f084d3ba329ce6dcbb82e65cbcabaa5b3ae6573f5b85fec4841080c31c55"}
Jul 30 10:49:35.147820 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.147460    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:49:35.149752 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.149720    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spcnq" event={"ID":"5e588b42-d61a-4087-8a87-8b941a382ad2","Type":"ContainerStarted","Data":"a828aa6136a71a6ae35e2d58df81594567a560952bf8426e8a96c0f7eaf54681"}
Jul 30 10:49:35.151905 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.151882    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94" event={"ID":"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6","Type":"ContainerStarted","Data":"c246dd78ecae5dc10a4676c39985cd15d119151f228c1cd265bdf6e01f81564c"}
Jul 30 10:49:35.152043 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.151925    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94" event={"ID":"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6","Type":"ContainerStarted","Data":"a11707fd59da6519eb3406181728f0ba2c7e1fc2f5621729105c351dd3e578dd"}
Jul 30 10:49:35.152043 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.151955    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94" event={"ID":"400e5d0e-2ec4-4ecd-b02a-af44fd3079f6","Type":"ContainerStarted","Data":"f81504cb707c85a422cd1aa9a0b867222533a4389574b7bc31ede59d250c5225"}
Jul 30 10:49:35.153648 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.153628    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4swtg" event={"ID":"8aeeb651-2172-4d17-bcb5-4e78a6b84278","Type":"ContainerStarted","Data":"e42834329e9bb505fcca39b712813727aeaa2a12bd229ebee8f333c9f60538aa"}
Jul 30 10:49:35.153717 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.153664    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4swtg" event={"ID":"8aeeb651-2172-4d17-bcb5-4e78a6b84278","Type":"ContainerStarted","Data":"b3153906dcf5490b3d54e34cde8971c1b82eb557035abf0af7c9d1dbec2e4d52"}
Jul 30 10:49:35.153717 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.153706    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4swtg"
Jul 30 10:49:35.154962 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.154944    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl" event={"ID":"661a2c0b-95c5-4e58-ac6e-e9a7ca12f430","Type":"ContainerStarted","Data":"dc96269ab5745617ec3af2713867e472571dda4b5dc616ae99a493f5609aa841"}
Jul 30 10:49:35.155072 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.155058    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:35.156488 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.156468    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cbrck" event={"ID":"20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d","Type":"ContainerStarted","Data":"cbd50e74d34ac4af1736061bf4c324205d8ce0f7c72047db5a96cb9952fd2b7a"}
Jul 30 10:49:35.157052 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.157035    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl"
Jul 30 10:49:35.171824 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.171781    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5qwx4" podStartSLOduration=79.15934635 podStartE2EDuration="1m21.171768937s" totalImagesPullingTime="2.012422587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:25.125056102 +0000 UTC m=+70.845940364" watchObservedRunningTime="2026-07-30 10:49:35.171768937 +0000 UTC m=+80.892653197"
Jul 30 10:49:35.172024 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.172005    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x72dc" podStartSLOduration=69.091048096 podStartE2EDuration="1m21.171999206s" totalImagesPullingTime="12.08095111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:35.170416865 +0000 UTC m=+80.891301129" watchObservedRunningTime="2026-07-30 10:49:35.171999206 +0000 UTC m=+80.892883511"
Jul 30 10:49:35.191121 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.191057    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66d5b6689f-lzb94" podStartSLOduration=3.158684114 podStartE2EDuration="15.191040214s" totalImagesPullingTime="12.0323561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:35.190909315 +0000 UTC m=+80.911793575" watchObservedRunningTime="2026-07-30 10:49:35.191040214 +0000 UTC m=+80.911924471"
Jul 30 10:49:35.213993 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.213942    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4swtg" podStartSLOduration=34.271250895 podStartE2EDuration="46.213926434s" totalImagesPullingTime="11.942675539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:35.212806489 +0000 UTC m=+80.933690751" watchObservedRunningTime="2026-07-30 10:49:35.213926434 +0000 UTC m=+80.934810696"
Jul 30 10:49:35.239921 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.239869    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-spcnq" podStartSLOduration=17.319063341 podStartE2EDuration="38.239852618s" totalImagesPullingTime="20.920789277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:35.239841052 +0000 UTC m=+80.960725313" watchObservedRunningTime="2026-07-30 10:49:35.239852618 +0000 UTC m=+80.960736876"
Jul 30 10:49:35.258384 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.258322    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cbrck" podStartSLOduration=69.265646805 podStartE2EDuration="1m21.258299687s" totalImagesPullingTime="11.992652882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:35.258289821 +0000 UTC m=+80.979174082" watchObservedRunningTime="2026-07-30 10:49:35.258299687 +0000 UTC m=+80.979183942"
Jul 30 10:49:35.278122 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:35.278058    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-644b68b5f-52nxl" podStartSLOduration=3.2654124429999998 podStartE2EDuration="15.27804119s" totalImagesPullingTime="12.012628747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:35.27754044 +0000 UTC m=+80.998424711" watchObservedRunningTime="2026-07-30 10:49:35.27804119 +0000 UTC m=+80.998925451"
Jul 30 10:49:37.908294 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:37.908178    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:39.646378 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:39.646342    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:39.703110 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:39.703078    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:39.854372 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:39.854337    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:40.069810 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.069781    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:40.132287 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.132254    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zvmqd"]
Jul 30 10:49:40.167777 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.167729    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:40.240528 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.240490    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:40.299003 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.298938    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:40.322955 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.322870    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.325758 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.325731    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r6g6q\""
Jul 30 10:49:40.327004 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.326978    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 30 10:49:40.327165 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.327069    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 30 10:49:40.327258 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.327166    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 30 10:49:40.327258 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.327209    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 30 10:49:40.327342 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.326978    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 30 10:49:40.327430 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.327414    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 30 10:49:40.369961 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.369921    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:40.451003 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.450967    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77f44fe7-44bd-404c-9845-bb065387ce70-metrics-client-ca\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.451003 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.451002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-tls\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.451247 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.451023    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77f44fe7-44bd-404c-9845-bb065387ce70-root\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.451247 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.451040    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-textfile\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.451247 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.451065    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwwfk\" (UniqueName: \"kubernetes.io/projected/77f44fe7-44bd-404c-9845-bb065387ce70-kube-api-access-jwwfk\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.451247 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.451118    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.451247 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.451163    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-wtmp\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.451247 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.451199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.451247 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.451224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77f44fe7-44bd-404c-9845-bb065387ce70-sys\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.511509 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.511459    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:40.552503 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-wtmp\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.552664 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.552664 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552550    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77f44fe7-44bd-404c-9845-bb065387ce70-sys\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.552664 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552583    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77f44fe7-44bd-404c-9845-bb065387ce70-metrics-client-ca\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.552664 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552604    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-tls\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.552664 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552626    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77f44fe7-44bd-404c-9845-bb065387ce70-root\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.552898 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552666    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77f44fe7-44bd-404c-9845-bb065387ce70-sys\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.552898 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552676    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77f44fe7-44bd-404c-9845-bb065387ce70-root\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.552898 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552692    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-textfile\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.552898 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552738    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwwfk\" (UniqueName: \"kubernetes.io/projected/77f44fe7-44bd-404c-9845-bb065387ce70-kube-api-access-jwwfk\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.552898 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.552781    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.553174 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.553011    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-textfile\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.553174 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.553160    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-wtmp\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.553284 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.553227    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.553393 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.553370    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77f44fe7-44bd-404c-9845-bb065387ce70-metrics-client-ca\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.555182 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.555157    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-tls\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.555280 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.555186    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77f44fe7-44bd-404c-9845-bb065387ce70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.565758 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.565739    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwwfk\" (UniqueName: \"kubernetes.io/projected/77f44fe7-44bd-404c-9845-bb065387ce70-kube-api-access-jwwfk\") pod \"node-exporter-zvmqd\" (UID: \"77f44fe7-44bd-404c-9845-bb065387ce70\") " pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.582298 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.582229    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:40.635459 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.635432    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zvmqd"
Jul 30 10:49:40.674384 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:49:40.674350    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f44fe7_44bd_404c_9845_bb065387ce70.slice/crio-4526547fde4368bfe05f74fadf3b77bf1b139bc0b4835f5e1910c30faf3e030e WatchSource:0}: Error finding container 4526547fde4368bfe05f74fadf3b77bf1b139bc0b4835f5e1910c30faf3e030e: Status 404 returned error can't find the container with id 4526547fde4368bfe05f74fadf3b77bf1b139bc0b4835f5e1910c30faf3e030e
Jul 30 10:49:40.840333 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.840245    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:40.920941 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:40.920908    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:41.081201 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:41.081158    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:41.178110 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:41.178070    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvmqd" event={"ID":"77f44fe7-44bd-404c-9845-bb065387ce70","Type":"ContainerStarted","Data":"4526547fde4368bfe05f74fadf3b77bf1b139bc0b4835f5e1910c30faf3e030e"}
Jul 30 10:49:41.181778 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:41.181752    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:41.304656 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:41.304620    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:41.599906 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:41.599824    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:43.069386 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:43.069351    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:43.166620 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:43.166585    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:44.001883 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:44.001849    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:44.063314 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:44.063285    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:44.128723 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:44.128692    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:44.188947 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:44.188913    2571 generic.go:356] "Generic (PLEG): container finished" podID="77f44fe7-44bd-404c-9845-bb065387ce70" containerID="5ffe200f80ce0a118137ac76ce0c2c2ba3bd293fb1af85abdf9cffab322f7405" exitCode=0
Jul 30 10:49:44.189086 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:44.188980    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvmqd" event={"ID":"77f44fe7-44bd-404c-9845-bb065387ce70","Type":"ContainerDied","Data":"5ffe200f80ce0a118137ac76ce0c2c2ba3bd293fb1af85abdf9cffab322f7405"}
Jul 30 10:49:44.292389 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:44.292361    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:44.395164 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:44.395142    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:44.884545 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:44.884511    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:44.951036 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:44.951007    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:45.160067 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:45.159974    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:45.162390 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:45.162367    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4swtg"
Jul 30 10:49:45.194087 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:45.194057    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvmqd" event={"ID":"77f44fe7-44bd-404c-9845-bb065387ce70","Type":"ContainerStarted","Data":"7f898c132453d8e1b2789b38e31c9d2c7314a77dcb5d107cdf04671f68405c85"}
Jul 30 10:49:45.194225 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:45.194133    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvmqd" event={"ID":"77f44fe7-44bd-404c-9845-bb065387ce70","Type":"ContainerStarted","Data":"d7953ff9f564f7f894549576c906033d16a840f77372cf2f3a23cf26f46e1d06"}
Jul 30 10:49:45.220647 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:45.219160    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zvmqd" podStartSLOduration=2.446474881 podStartE2EDuration="5.219140691s" totalImagesPullingTime="2.685384394s" totalInitContainerRuntime="87.281416ms" isStatefulPod=true podCreationTimestamp="2026-07-30 10:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:45.216817227 +0000 UTC m=+90.937701490" watchObservedRunningTime="2026-07-30 10:49:45.219140691 +0000 UTC m=+90.940024953"
Jul 30 10:49:46.357452 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:46.357403    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:47.121517 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:47.121482    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:47.217031 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:47.216999    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:50.151236 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.151179    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:50.252409 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.252375    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:50.287133 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.287088    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7749dd7949-scsc6"]
Jul 30 10:49:50.306743 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.306717    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7749dd7949-scsc6"]
Jul 30 10:49:50.306882 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.306811    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.309971 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.309947    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 30 10:49:50.310125 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.309973    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 30 10:49:50.311255 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.311235    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 30 10:49:50.311506 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.311448    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 30 10:49:50.311592 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.311258    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 30 10:49:50.311592 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.311328    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 30 10:49:50.311680 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.311259    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-drg7x\""
Jul 30 10:49:50.312674 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.312650    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 30 10:49:50.318083 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.318058    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 30 10:49:50.320640 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.320621    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-config\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.320731 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.320654    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldwn\" (UniqueName: \"kubernetes.io/projected/0da1e482-449f-440d-bf04-17dcc81a1ef7-kube-api-access-nldwn\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.320731 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.320678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-trusted-ca-bundle\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.320731 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.320695    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-serving-cert\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.320825 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.320754    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-oauth-config\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.320825 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.320798    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-service-ca\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.320887 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.320827    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-oauth-serving-cert\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.421814 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.421728    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-oauth-serving-cert\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.421814 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.421777    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-config\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.421814 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.421794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldwn\" (UniqueName: \"kubernetes.io/projected/0da1e482-449f-440d-bf04-17dcc81a1ef7-kube-api-access-nldwn\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.421814 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.421814    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-trusted-ca-bundle\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.422151 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.421832    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-serving-cert\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.422151 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.421862    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-oauth-config\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.422151 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.421892    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-service-ca\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.422630 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.422597    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-config\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.422748 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.422653    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-oauth-serving-cert\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.422748 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.422659    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-service-ca\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.422865 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.422831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-trusted-ca-bundle\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.424399 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.424367    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-serving-cert\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.424530 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.424505    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-oauth-config\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.431070 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.431046    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldwn\" (UniqueName: \"kubernetes.io/projected/0da1e482-449f-440d-bf04-17dcc81a1ef7-kube-api-access-nldwn\") pod \"console-7749dd7949-scsc6\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") " pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.620389 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.620346    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:49:50.765613 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:50.765589    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7749dd7949-scsc6"]
Jul 30 10:49:50.767808 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:49:50.767786    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da1e482_449f_440d_bf04_17dcc81a1ef7.slice/crio-875df11d1cdd08b20ac1ef5dbadb521a550aefdbef62c2094e335d7be8a72792 WatchSource:0}: Error finding container 875df11d1cdd08b20ac1ef5dbadb521a550aefdbef62c2094e335d7be8a72792: Status 404 returned error can't find the container with id 875df11d1cdd08b20ac1ef5dbadb521a550aefdbef62c2094e335d7be8a72792
Jul 30 10:49:51.215763 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:51.215734    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7749dd7949-scsc6" event={"ID":"0da1e482-449f-440d-bf04-17dcc81a1ef7","Type":"ContainerStarted","Data":"875df11d1cdd08b20ac1ef5dbadb521a550aefdbef62c2094e335d7be8a72792"}
Jul 30 10:49:52.641475 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:52.641437    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:53.670602 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:53.670571    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:53.770169 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:53.770129    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:53.870175 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:53.870152    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:53.971051 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:53.971013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:54.228811 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:54.228727    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7749dd7949-scsc6" event={"ID":"0da1e482-449f-440d-bf04-17dcc81a1ef7","Type":"ContainerStarted","Data":"00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10"}
Jul 30 10:49:54.248723 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:54.248674    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7749dd7949-scsc6" podStartSLOduration=1.163580402 podStartE2EDuration="4.248657159s" totalImagesPullingTime="3.085076757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:54.248000303 +0000 UTC m=+99.968884567" watchObservedRunningTime="2026-07-30 10:49:54.248657159 +0000 UTC m=+99.969541420"
Jul 30 10:49:56.095230 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:56.095191    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:56.194081 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:56.194043    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:56.651776 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:56.651723    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:49:57.256206 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:49:57.256175    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:50:00.620929 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:00.620890    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:50:00.621376 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:00.620978    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:50:00.625840 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:00.625817    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:50:01.254950 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:01.254924    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:50:01.450683 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:01.450617    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:50:01.552924 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:01.552821    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:50:01.653885 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:01.653853    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:50:02.104569 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:02.104526    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:50:02.164148 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:02.164116    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:50:06.163507 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:06.163474    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x72dc"
Jul 30 10:50:17.134548 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:17.134508    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:50:17.683388 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:17.683354    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:50:59.361323 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:59.361286    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:50:59.578634 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:50:59.578594    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:51:08.300392 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.300315    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f568b9748-z8nqh"]
Jul 30 10:51:08.303230 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.303211    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.313587 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.313564    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f568b9748-z8nqh"]
Jul 30 10:51:08.359856 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.359827    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:51:08.424425 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.424401    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:51:08.454340 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.454317    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-trusted-ca-bundle\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.454446 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.454348    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-serving-cert\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.454446 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.454383    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-oauth-config\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.454446 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.454402    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-service-ca\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.454446 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.454426    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-oauth-serving-cert\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.454446 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.454443    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2g6x\" (UniqueName: \"kubernetes.io/projected/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-kube-api-access-s2g6x\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.454628 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.454462    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-config\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.555480 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.555406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-serving-cert\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.555480 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.555444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-oauth-config\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.555480 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.555461    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-service-ca\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.555716 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.555487    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-oauth-serving-cert\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.555716 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.555511    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2g6x\" (UniqueName: \"kubernetes.io/projected/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-kube-api-access-s2g6x\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.555716 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.555536    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-config\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.555716 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.555603    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-trusted-ca-bundle\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.556360 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.556334    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-config\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.556360 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.556353    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-oauth-serving-cert\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.556488 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.556378    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-service-ca\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.556655 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.556633    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-trusted-ca-bundle\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.557973 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.557950    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-serving-cert\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.558041 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.557977    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-oauth-config\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.563594 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.563577    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2g6x\" (UniqueName: \"kubernetes.io/projected/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-kube-api-access-s2g6x\") pod \"console-5f568b9748-z8nqh\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") " pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.611477 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.611457    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:08.739277 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:08.739160    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f568b9748-z8nqh"]
Jul 30 10:51:08.741868 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:51:08.741842    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca1c3fc_da2d_4ebd_91c3_a6c88a5b0e2b.slice/crio-0db2578d81fa2402937aba9c19f42afe12d723c98b617a424b9a943bcb995c3f WatchSource:0}: Error finding container 0db2578d81fa2402937aba9c19f42afe12d723c98b617a424b9a943bcb995c3f: Status 404 returned error can't find the container with id 0db2578d81fa2402937aba9c19f42afe12d723c98b617a424b9a943bcb995c3f
Jul 30 10:51:09.470084 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:09.470048    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f568b9748-z8nqh" event={"ID":"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b","Type":"ContainerStarted","Data":"ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054"}
Jul 30 10:51:09.470577 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:09.470123    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f568b9748-z8nqh" event={"ID":"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b","Type":"ContainerStarted","Data":"0db2578d81fa2402937aba9c19f42afe12d723c98b617a424b9a943bcb995c3f"}
Jul 30 10:51:09.491542 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:09.491496    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f568b9748-z8nqh" podStartSLOduration=1.491481964 podStartE2EDuration="1.491481964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:51:09.489192542 +0000 UTC m=+175.210076800" watchObservedRunningTime="2026-07-30 10:51:09.491481964 +0000 UTC m=+175.212366225"
Jul 30 10:51:18.612247 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:18.612196    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:18.612674 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:18.612272    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:18.616950 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:18.616928    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:19.506981 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:19.506950    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:51:19.564198 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:19.564170    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7749dd7949-scsc6"]
Jul 30 10:51:19.670488 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:19.670458    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:51:19.732783 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:19.732750    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:51:19.786807 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:19.786725    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:51:32.157049 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:32.157014    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:51:44.583457 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.583412    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7749dd7949-scsc6" podUID="0da1e482-449f-440d-bf04-17dcc81a1ef7" containerName="console" containerID="cri-o://00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10" gracePeriod=15
Jul 30 10:51:44.817473 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.817449    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:51:44.912229 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912147    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-trusted-ca-bundle\") pod \"0da1e482-449f-440d-bf04-17dcc81a1ef7\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") "
Jul 30 10:51:44.912229 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912183    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-oauth-serving-cert\") pod \"0da1e482-449f-440d-bf04-17dcc81a1ef7\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") "
Jul 30 10:51:44.912229 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912213    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-serving-cert\") pod \"0da1e482-449f-440d-bf04-17dcc81a1ef7\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") "
Jul 30 10:51:44.912450 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912244    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nldwn\" (UniqueName: \"kubernetes.io/projected/0da1e482-449f-440d-bf04-17dcc81a1ef7-kube-api-access-nldwn\") pod \"0da1e482-449f-440d-bf04-17dcc81a1ef7\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") "
Jul 30 10:51:44.912450 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912300    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-oauth-config\") pod \"0da1e482-449f-440d-bf04-17dcc81a1ef7\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") "
Jul 30 10:51:44.912450 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912354    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-config\") pod \"0da1e482-449f-440d-bf04-17dcc81a1ef7\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") "
Jul 30 10:51:44.912450 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912378    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-service-ca\") pod \"0da1e482-449f-440d-bf04-17dcc81a1ef7\" (UID: \"0da1e482-449f-440d-bf04-17dcc81a1ef7\") "
Jul 30 10:51:44.912732 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912703    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0da1e482-449f-440d-bf04-17dcc81a1ef7" (UID: "0da1e482-449f-440d-bf04-17dcc81a1ef7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:51:44.912831 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912749    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-config" (OuterVolumeSpecName: "console-config") pod "0da1e482-449f-440d-bf04-17dcc81a1ef7" (UID: "0da1e482-449f-440d-bf04-17dcc81a1ef7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:51:44.912891 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912821    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-service-ca" (OuterVolumeSpecName: "service-ca") pod "0da1e482-449f-440d-bf04-17dcc81a1ef7" (UID: "0da1e482-449f-440d-bf04-17dcc81a1ef7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:51:44.912891 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.912832    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0da1e482-449f-440d-bf04-17dcc81a1ef7" (UID: "0da1e482-449f-440d-bf04-17dcc81a1ef7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:51:44.914633 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.914598    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0da1e482-449f-440d-bf04-17dcc81a1ef7" (UID: "0da1e482-449f-440d-bf04-17dcc81a1ef7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 10:51:44.914848 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.914815    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0da1e482-449f-440d-bf04-17dcc81a1ef7" (UID: "0da1e482-449f-440d-bf04-17dcc81a1ef7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 10:51:44.914930 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:44.914843    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da1e482-449f-440d-bf04-17dcc81a1ef7-kube-api-access-nldwn" (OuterVolumeSpecName: "kube-api-access-nldwn") pod "0da1e482-449f-440d-bf04-17dcc81a1ef7" (UID: "0da1e482-449f-440d-bf04-17dcc81a1ef7"). InnerVolumeSpecName "kube-api-access-nldwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 10:51:45.013281 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.013227    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-config\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:51:45.013281 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.013274    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:51:45.013281 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.013287    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-trusted-ca-bundle\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:51:45.013516 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.013300    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0da1e482-449f-440d-bf04-17dcc81a1ef7-oauth-serving-cert\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:51:45.013516 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.013314    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-serving-cert\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:51:45.013516 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.013325    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nldwn\" (UniqueName: \"kubernetes.io/projected/0da1e482-449f-440d-bf04-17dcc81a1ef7-kube-api-access-nldwn\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:51:45.013516 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.013338    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0da1e482-449f-440d-bf04-17dcc81a1ef7-console-oauth-config\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:51:45.589581 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.589537    2571 generic.go:356] "Generic (PLEG): container finished" podID="0da1e482-449f-440d-bf04-17dcc81a1ef7" containerID="00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10" exitCode=0
Jul 30 10:51:45.589968 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.589589    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7749dd7949-scsc6" event={"ID":"0da1e482-449f-440d-bf04-17dcc81a1ef7","Type":"ContainerDied","Data":"00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10"}
Jul 30 10:51:45.589968 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.589609    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7749dd7949-scsc6"
Jul 30 10:51:45.589968 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.589641    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7749dd7949-scsc6" event={"ID":"0da1e482-449f-440d-bf04-17dcc81a1ef7","Type":"ContainerDied","Data":"875df11d1cdd08b20ac1ef5dbadb521a550aefdbef62c2094e335d7be8a72792"}
Jul 30 10:51:45.589968 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.589670    2571 scope.go:117] "RemoveContainer" containerID="00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10"
Jul 30 10:51:45.598458 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.598441    2571 scope.go:117] "RemoveContainer" containerID="00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10"
Jul 30 10:51:45.598728 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:51:45.598701    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10\": container with ID starting with 00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10 not found: ID does not exist" containerID="00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10"
Jul 30 10:51:45.598776 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.598736    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10"} err="failed to get container status \"00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10\": rpc error: code = NotFound desc = could not find container \"00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10\": container with ID starting with 00475c4355aa77c0528887469d1a11b6080402ab2e44e4824c0c28105958da10 not found: ID does not exist"
Jul 30 10:51:45.609264 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.609237    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7749dd7949-scsc6"]
Jul 30 10:51:45.612555 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:45.612533    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7749dd7949-scsc6"]
Jul 30 10:51:46.853348 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:46.853315    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da1e482-449f-440d-bf04-17dcc81a1ef7" path="/var/lib/kubelet/pods/0da1e482-449f-440d-bf04-17dcc81a1ef7/volumes"
Jul 30 10:51:48.170678 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:48.170644    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:51:52.923972 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:52.923900    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:51:53.009910 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:51:53.009874    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:11.003670 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:11.003626    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:14.712361 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:14.712328    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:14.812558 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:14.812523    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:26.759757 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:26.759704    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:26.860267 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:26.860198    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:29.556041 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.553483    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fbfb7c986-b5dgg"]
Jul 30 10:52:29.556041 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.553697    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1e482-449f-440d-bf04-17dcc81a1ef7" containerName="console"
Jul 30 10:52:29.556041 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.553707    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1e482-449f-440d-bf04-17dcc81a1ef7" containerName="console"
Jul 30 10:52:29.556041 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.553756    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da1e482-449f-440d-bf04-17dcc81a1ef7" containerName="console"
Jul 30 10:52:29.557755 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.557737    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.570146 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.570120    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbfb7c986-b5dgg"]
Jul 30 10:52:29.607552 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.607524    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-oauth-serving-cert\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.607658 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.607561    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65xp2\" (UniqueName: \"kubernetes.io/projected/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-kube-api-access-65xp2\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.607658 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.607589    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-trusted-ca-bundle\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.607658 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.607631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-config\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.607766 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.607663    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-oauth-config\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.607766 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.607686    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-serving-cert\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.607766 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.607713    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-service-ca\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.639802 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.639776    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:29.709031 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65xp2\" (UniqueName: \"kubernetes.io/projected/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-kube-api-access-65xp2\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.709152 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-trusted-ca-bundle\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.709152 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709074    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-config\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.709152 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709094    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-oauth-config\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.709152 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709137    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-serving-cert\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.709342 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709164    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-service-ca\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.709342 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709201    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-oauth-serving-cert\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.709833 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709810    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-config\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.709951 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709907    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-oauth-serving-cert\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.710016 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709946    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-service-ca\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.710016 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.709983    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-trusted-ca-bundle\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.711584 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.711553    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-oauth-config\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.711739 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.711713    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-serving-cert\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.717993 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.717974    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65xp2\" (UniqueName: \"kubernetes.io/projected/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-kube-api-access-65xp2\") pod \"console-7fbfb7c986-b5dgg\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") " pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.867214 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.867135    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:29.990458 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:29.990427    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbfb7c986-b5dgg"]
Jul 30 10:52:29.993323 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:52:29.993296    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a16f59a_a67e_4c0d_a7b2_f31149946a7c.slice/crio-92dee2c043f376cdc2af97004dca566f667f47697e9bb34f0f81aa61a75635ca WatchSource:0}: Error finding container 92dee2c043f376cdc2af97004dca566f667f47697e9bb34f0f81aa61a75635ca: Status 404 returned error can't find the container with id 92dee2c043f376cdc2af97004dca566f667f47697e9bb34f0f81aa61a75635ca
Jul 30 10:52:30.741025 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:30.740990    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbfb7c986-b5dgg" event={"ID":"6a16f59a-a67e-4c0d-a7b2-f31149946a7c","Type":"ContainerStarted","Data":"9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4"}
Jul 30 10:52:30.741430 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:30.741065    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbfb7c986-b5dgg" event={"ID":"6a16f59a-a67e-4c0d-a7b2-f31149946a7c","Type":"ContainerStarted","Data":"92dee2c043f376cdc2af97004dca566f667f47697e9bb34f0f81aa61a75635ca"}
Jul 30 10:52:30.759404 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:30.759358    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fbfb7c986-b5dgg" podStartSLOduration=1.759344965 podStartE2EDuration="1.759344965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:52:30.757789326 +0000 UTC m=+256.478673588" watchObservedRunningTime="2026-07-30 10:52:30.759344965 +0000 UTC m=+256.480229278"
Jul 30 10:52:32.103022 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:32.102983    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:32.204813 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:32.204780    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:32.302646 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:32.302619    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:39.867754 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:39.867661    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:39.868193 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:39.867767    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:39.872564 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:39.872543    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:40.776815 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:40.776780    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:52:40.822961 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:40.822932    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f568b9748-z8nqh"]
Jul 30 10:52:40.938145 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:40.938115    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:52:41.125148 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:52:41.125053    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:05.841250 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:05.841188    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f568b9748-z8nqh" podUID="cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" containerName="console" containerID="cri-o://ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054" gracePeriod=15
Jul 30 10:53:06.072811 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.072787    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:53:06.164226 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164153    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-oauth-config\") pod \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") "
Jul 30 10:53:06.164226 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164190    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-serving-cert\") pod \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") "
Jul 30 10:53:06.164226 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164223    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2g6x\" (UniqueName: \"kubernetes.io/projected/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-kube-api-access-s2g6x\") pod \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") "
Jul 30 10:53:06.164498 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164243    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-service-ca\") pod \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") "
Jul 30 10:53:06.164498 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164263    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-oauth-serving-cert\") pod \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") "
Jul 30 10:53:06.164498 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164279    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-config\") pod \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") "
Jul 30 10:53:06.164498 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164301    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-trusted-ca-bundle\") pod \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\" (UID: \"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b\") "
Jul 30 10:53:06.164784 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164737    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-config" (OuterVolumeSpecName: "console-config") pod "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" (UID: "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:53:06.164784 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164770    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-service-ca" (OuterVolumeSpecName: "service-ca") pod "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" (UID: "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:53:06.164899 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164764    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" (UID: "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:53:06.164899 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.164792    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" (UID: "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:53:06.166443 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.166416    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-kube-api-access-s2g6x" (OuterVolumeSpecName: "kube-api-access-s2g6x") pod "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" (UID: "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b"). InnerVolumeSpecName "kube-api-access-s2g6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 10:53:06.166768 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.166740    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" (UID: "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 10:53:06.166768 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.166754    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" (UID: "cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 10:53:06.265240 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.265207    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-oauth-config\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:53:06.265240 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.265234    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-serving-cert\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:53:06.265240 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.265243    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2g6x\" (UniqueName: \"kubernetes.io/projected/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-kube-api-access-s2g6x\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:53:06.265457 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.265252    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:53:06.265457 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.265260    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-oauth-serving-cert\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:53:06.265457 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.265269    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-console-config\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:53:06.265457 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.265277    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b-trusted-ca-bundle\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:53:06.856476 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.856441    2571 generic.go:356] "Generic (PLEG): container finished" podID="cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" containerID="ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054" exitCode=0
Jul 30 10:53:06.856476 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.856476    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f568b9748-z8nqh"
Jul 30 10:53:06.856476 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.856481    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f568b9748-z8nqh" event={"ID":"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b","Type":"ContainerDied","Data":"ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054"}
Jul 30 10:53:06.856993 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.856522    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f568b9748-z8nqh" event={"ID":"cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b","Type":"ContainerDied","Data":"0db2578d81fa2402937aba9c19f42afe12d723c98b617a424b9a943bcb995c3f"}
Jul 30 10:53:06.856993 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.856548    2571 scope.go:117] "RemoveContainer" containerID="ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054"
Jul 30 10:53:06.867332 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.867305    2571 scope.go:117] "RemoveContainer" containerID="ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054"
Jul 30 10:53:06.867617 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:53:06.867594    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054\": container with ID starting with ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054 not found: ID does not exist" containerID="ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054"
Jul 30 10:53:06.867673 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.867625    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054"} err="failed to get container status \"ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054\": rpc error: code = NotFound desc = could not find container \"ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054\": container with ID starting with ee7efdd1181b074e65875512aa8ffd9b9d25a00b9fbf32e9b6c12d07527a9054 not found: ID does not exist"
Jul 30 10:53:06.880029 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.880005    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f568b9748-z8nqh"]
Jul 30 10:53:06.883296 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:06.883275    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f568b9748-z8nqh"]
Jul 30 10:53:08.853718 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:08.853685    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" path="/var/lib/kubelet/pods/cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b/volumes"
Jul 30 10:53:14.664138 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:14.664116    2571 kubelet.go:1648] "Image garbage collection succeeded"
Jul 30 10:53:22.699241 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:22.699202    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:24.546734 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:24.546699    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:49.259663 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:49.259628    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:50.600772 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:50.600734    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:52.872162 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:52.872128    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:52.968617 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:52.968587    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:54.516434 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:54.516399    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:54.640275 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:54.640244    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:55.006113 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:55.006068    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:57.363227 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:57.363194    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:57.464023 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:57.463988    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:57.807085 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:57.807057    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:59.850207 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:59.850175    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:53:59.900913 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:53:59.900878    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:04.608274 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:04.608237    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:04.678361 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:04.678325    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:04.814180 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:04.814145    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:05.475835 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.475802    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tw5cm"]
Jul 30 10:54:05.476033 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.476020    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" containerName="console"
Jul 30 10:54:05.476084 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.476036    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" containerName="console"
Jul 30 10:54:05.476084 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.476080    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca1c3fc-da2d-4ebd-91c3-a6c88a5b0e2b" containerName="console"
Jul 30 10:54:05.478773 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.478757    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tw5cm"
Jul 30 10:54:05.481873 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.481857    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 30 10:54:05.482975 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.482958    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-szhwq\""
Jul 30 10:54:05.483029 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.482962    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 30 10:54:05.489310 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.489288    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tw5cm"]
Jul 30 10:54:05.555916 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.555888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69551b9e-a6bd-4805-8efd-d5e6163dac38-bound-sa-token\") pod \"cert-manager-55b4d954d5-tw5cm\" (UID: \"69551b9e-a6bd-4805-8efd-d5e6163dac38\") " pod="cert-manager/cert-manager-55b4d954d5-tw5cm"
Jul 30 10:54:05.556050 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.555924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp2rs\" (UniqueName: \"kubernetes.io/projected/69551b9e-a6bd-4805-8efd-d5e6163dac38-kube-api-access-qp2rs\") pod \"cert-manager-55b4d954d5-tw5cm\" (UID: \"69551b9e-a6bd-4805-8efd-d5e6163dac38\") " pod="cert-manager/cert-manager-55b4d954d5-tw5cm"
Jul 30 10:54:05.656747 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.656719    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69551b9e-a6bd-4805-8efd-d5e6163dac38-bound-sa-token\") pod \"cert-manager-55b4d954d5-tw5cm\" (UID: \"69551b9e-a6bd-4805-8efd-d5e6163dac38\") " pod="cert-manager/cert-manager-55b4d954d5-tw5cm"
Jul 30 10:54:05.657079 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.656760    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp2rs\" (UniqueName: \"kubernetes.io/projected/69551b9e-a6bd-4805-8efd-d5e6163dac38-kube-api-access-qp2rs\") pod \"cert-manager-55b4d954d5-tw5cm\" (UID: \"69551b9e-a6bd-4805-8efd-d5e6163dac38\") " pod="cert-manager/cert-manager-55b4d954d5-tw5cm"
Jul 30 10:54:05.665630 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.665608    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69551b9e-a6bd-4805-8efd-d5e6163dac38-bound-sa-token\") pod \"cert-manager-55b4d954d5-tw5cm\" (UID: \"69551b9e-a6bd-4805-8efd-d5e6163dac38\") " pod="cert-manager/cert-manager-55b4d954d5-tw5cm"
Jul 30 10:54:05.665909 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.665889    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp2rs\" (UniqueName: \"kubernetes.io/projected/69551b9e-a6bd-4805-8efd-d5e6163dac38-kube-api-access-qp2rs\") pod \"cert-manager-55b4d954d5-tw5cm\" (UID: \"69551b9e-a6bd-4805-8efd-d5e6163dac38\") " pod="cert-manager/cert-manager-55b4d954d5-tw5cm"
Jul 30 10:54:05.787854 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.787773    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tw5cm"
Jul 30 10:54:05.909745 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.909617    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tw5cm"]
Jul 30 10:54:05.912292 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:54:05.912266    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69551b9e_a6bd_4805_8efd_d5e6163dac38.slice/crio-c551883ecc0136187f3a7a3ff90104445b29c7132695d1ebbc3972b1ace04286 WatchSource:0}: Error finding container c551883ecc0136187f3a7a3ff90104445b29c7132695d1ebbc3972b1ace04286: Status 404 returned error can't find the container with id c551883ecc0136187f3a7a3ff90104445b29c7132695d1ebbc3972b1ace04286
Jul 30 10:54:05.914312 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:05.914292    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 10:54:06.038602 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:06.038536    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tw5cm" event={"ID":"69551b9e-a6bd-4805-8efd-d5e6163dac38","Type":"ContainerStarted","Data":"c551883ecc0136187f3a7a3ff90104445b29c7132695d1ebbc3972b1ace04286"}
Jul 30 10:54:07.527238 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:07.527204    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:07.628289 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:07.628237    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:09.050513 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:09.050471    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tw5cm" event={"ID":"69551b9e-a6bd-4805-8efd-d5e6163dac38","Type":"ContainerStarted","Data":"f75ec8fdd7232b8782e33e0ec9c49d81f7b89c79d6090fc8b0af6e390940a195"}
Jul 30 10:54:09.068305 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:09.068262    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-tw5cm" podStartSLOduration=1.045844829 podStartE2EDuration="4.068248009s" totalImagesPullingTime="3.02240318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:09.068118993 +0000 UTC m=+354.789003251" watchObservedRunningTime="2026-07-30 10:54:09.068248009 +0000 UTC m=+354.789132270"
Jul 30 10:54:13.932639 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:13.932608    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:14.035909 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:14.035875    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:15.821497 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:15.821461    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:27.891092 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:27.891051    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:27.991313 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:27.991280    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:28.523495 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:28.523460    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:29.019523 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.019489    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:29.129757 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.129727    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:29.134740 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.134711    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"]
Jul 30 10:54:29.137945 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.137925    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.140890 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.140868    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 30 10:54:29.141313 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.141290    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wx6nc\""
Jul 30 10:54:29.141313 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.141302    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 30 10:54:29.141313 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.141296    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 30 10:54:29.141522 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.141333    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 30 10:54:29.154671 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.154643    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"]
Jul 30 10:54:29.217419 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.217385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmsk5\" (UniqueName: \"kubernetes.io/projected/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-kube-api-access-xmsk5\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.217574 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.217432    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.217574 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.217506    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-tmp\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.217574 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.217563    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-manifests\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.217725 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.217589    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.257899 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.257855    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:29.318255 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.318176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-manifests\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.318255 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.318208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.318435 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.318345    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmsk5\" (UniqueName: \"kubernetes.io/projected/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-kube-api-access-xmsk5\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.318435 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.318397    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.318515 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.318452    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-tmp\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.318575 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.318557    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-manifests\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.320672 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.320640    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-tmp\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.320806 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.320738    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.320806 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.320772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.339823 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.339795    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmsk5\" (UniqueName: \"kubernetes.io/projected/cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba-kube-api-access-xmsk5\") pod \"opendatahub-operator-controller-manager-6d97895fdd-4dqwm\" (UID: \"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.362601 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.362574    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:29.448523 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.448492    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:29.460666 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.460635    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:29.567783 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.567735    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:29.583288 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.583267    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"]
Jul 30 10:54:29.585770 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:54:29.585743    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb1a3d1_f6b0_4bbb_af9b_8d9b14e574ba.slice/crio-9bb0ee23ec8d6546c1579d2c42dd3960ac1788eda8f718fc809af8b6250e5d8c WatchSource:0}: Error finding container 9bb0ee23ec8d6546c1579d2c42dd3960ac1788eda8f718fc809af8b6250e5d8c: Status 404 returned error can't find the container with id 9bb0ee23ec8d6546c1579d2c42dd3960ac1788eda8f718fc809af8b6250e5d8c
Jul 30 10:54:29.657432 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.657377    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:29.757082 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.757040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:29.857704 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:29.857621    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:30.123930 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:30.123851    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm" event={"ID":"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba","Type":"ContainerStarted","Data":"9bb0ee23ec8d6546c1579d2c42dd3960ac1788eda8f718fc809af8b6250e5d8c"}
Jul 30 10:54:32.094767 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:32.094724    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:32.202855 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:32.202823    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:33.137706 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:33.137667    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm" event={"ID":"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba","Type":"ContainerStarted","Data":"4c3405081bbf480585536841999735567166b9fc9afb8f2e66877b4eae154055"}
Jul 30 10:54:34.142881 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:34.142794    2571 generic.go:356] "Generic (PLEG): container finished" podID="cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba" containerID="4c3405081bbf480585536841999735567166b9fc9afb8f2e66877b4eae154055" exitCode=0
Jul 30 10:54:34.142881 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:34.142852    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm" event={"ID":"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba","Type":"ContainerDied","Data":"4c3405081bbf480585536841999735567166b9fc9afb8f2e66877b4eae154055"}
Jul 30 10:54:35.147934 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:35.147896    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm" event={"ID":"cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba","Type":"ContainerStarted","Data":"ece6af155777eb0a89a460e248de0c1a442d6972c647550b75cfca905464e976"}
Jul 30 10:54:35.148329 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:35.148008    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:35.171882 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:35.171833    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm" podStartSLOduration=1.6319269840000001 podStartE2EDuration="6.171817085s" totalImagesPullingTime="3.503588443s" totalInitContainerRuntime="1.036301658s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:54:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:35.169524551 +0000 UTC m=+380.890408811" watchObservedRunningTime="2026-07-30 10:54:35.171817085 +0000 UTC m=+380.892701345"
Jul 30 10:54:40.333048 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:40.333000    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:40.764436 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:40.764395    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:40.874538 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:40.874501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:46.154207 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:46.154172    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-4dqwm"
Jul 30 10:54:47.921115 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:47.921051    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:48.035254 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.035211    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:48.508054 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.508024    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:48.606905 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.606872    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"]
Jul 30 10:54:48.613614 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.613591    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:48.616652 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.616624    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 30 10:54:48.616770 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.616631    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 30 10:54:48.617766 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.617751    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 30 10:54:48.617870 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.617783    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 30 10:54:48.617919 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.617897    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k8twm\""
Jul 30 10:54:48.621644 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.621574    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"]
Jul 30 10:54:48.655018 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.654990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsld2\" (UniqueName: \"kubernetes.io/projected/de38d89b-a26e-4d4d-999f-00a1585e9484-kube-api-access-nsld2\") pod \"kube-auth-proxy-8674569cb6-mc94c\" (UID: \"de38d89b-a26e-4d4d-999f-00a1585e9484\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:48.655170 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.655051    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de38d89b-a26e-4d4d-999f-00a1585e9484-tmp\") pod \"kube-auth-proxy-8674569cb6-mc94c\" (UID: \"de38d89b-a26e-4d4d-999f-00a1585e9484\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:48.655170 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.655080    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de38d89b-a26e-4d4d-999f-00a1585e9484-tls-certs\") pod \"kube-auth-proxy-8674569cb6-mc94c\" (UID: \"de38d89b-a26e-4d4d-999f-00a1585e9484\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:48.662196 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.662173    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"]
Jul 30 10:54:48.664022 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.664008    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.666643 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.666597    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fq5bv\""
Jul 30 10:54:48.666643 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.666629    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 30 10:54:48.666808 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.666676    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 30 10:54:48.666855 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.666822    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 30 10:54:48.667042 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.667023    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 30 10:54:48.667123 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.667024    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 30 10:54:48.676935 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.676913    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"]
Jul 30 10:54:48.739903 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.739873    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:48.755980 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.755952    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de38d89b-a26e-4d4d-999f-00a1585e9484-tmp\") pod \"kube-auth-proxy-8674569cb6-mc94c\" (UID: \"de38d89b-a26e-4d4d-999f-00a1585e9484\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:48.756131 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.755986    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de38d89b-a26e-4d4d-999f-00a1585e9484-tls-certs\") pod \"kube-auth-proxy-8674569cb6-mc94c\" (UID: \"de38d89b-a26e-4d4d-999f-00a1585e9484\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:48.756131 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.756031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dqz8\" (UniqueName: \"kubernetes.io/projected/32a98e99-6467-42f3-9c62-e73f43e48a4b-kube-api-access-5dqz8\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.756131 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.756066    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32a98e99-6467-42f3-9c62-e73f43e48a4b-metrics-cert\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.756306 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:54:48.756188    2571 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 30 10:54:48.756306 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.756217    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsld2\" (UniqueName: \"kubernetes.io/projected/de38d89b-a26e-4d4d-999f-00a1585e9484-kube-api-access-nsld2\") pod \"kube-auth-proxy-8674569cb6-mc94c\" (UID: \"de38d89b-a26e-4d4d-999f-00a1585e9484\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:48.756306 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:54:48.756254    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de38d89b-a26e-4d4d-999f-00a1585e9484-tls-certs podName:de38d89b-a26e-4d4d-999f-00a1585e9484 nodeName:}" failed. No retries permitted until 2026-07-30 10:54:49.256239062 +0000 UTC m=+394.977123305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/de38d89b-a26e-4d4d-999f-00a1585e9484-tls-certs") pod "kube-auth-proxy-8674569cb6-mc94c" (UID: "de38d89b-a26e-4d4d-999f-00a1585e9484") : secret "kube-auth-proxy-tls" not found
Jul 30 10:54:48.756306 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.756276    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32a98e99-6467-42f3-9c62-e73f43e48a4b-cert\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.756517 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.756313    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32a98e99-6467-42f3-9c62-e73f43e48a4b-manager-config\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.758296 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.758250    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de38d89b-a26e-4d4d-999f-00a1585e9484-tmp\") pod \"kube-auth-proxy-8674569cb6-mc94c\" (UID: \"de38d89b-a26e-4d4d-999f-00a1585e9484\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:48.770878 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.770854    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsld2\" (UniqueName: \"kubernetes.io/projected/de38d89b-a26e-4d4d-999f-00a1585e9484-kube-api-access-nsld2\") pod \"kube-auth-proxy-8674569cb6-mc94c\" (UID: \"de38d89b-a26e-4d4d-999f-00a1585e9484\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:48.856902 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.856808    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32a98e99-6467-42f3-9c62-e73f43e48a4b-manager-config\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.857083 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.856968    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqz8\" (UniqueName: \"kubernetes.io/projected/32a98e99-6467-42f3-9c62-e73f43e48a4b-kube-api-access-5dqz8\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.857083 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.857010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32a98e99-6467-42f3-9c62-e73f43e48a4b-metrics-cert\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.857220 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.857088    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32a98e99-6467-42f3-9c62-e73f43e48a4b-cert\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.858751 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.857597    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32a98e99-6467-42f3-9c62-e73f43e48a4b-manager-config\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.862139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.861093    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32a98e99-6467-42f3-9c62-e73f43e48a4b-metrics-cert\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.862139 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.861122    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32a98e99-6467-42f3-9c62-e73f43e48a4b-cert\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.868533 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.868506    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqz8\" (UniqueName: \"kubernetes.io/projected/32a98e99-6467-42f3-9c62-e73f43e48a4b-kube-api-access-5dqz8\") pod \"lws-controller-manager-866774c595-tv5rb\" (UID: \"32a98e99-6467-42f3-9c62-e73f43e48a4b\") " pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:48.973311 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:48.973279    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:49.111505 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:49.111474    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"]
Jul 30 10:54:49.114867 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:54:49.114845    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a98e99_6467_42f3_9c62_e73f43e48a4b.slice/crio-1d5d29deaccc74ece02177505f8cfe79188a7a935a63e1c557925b6f926347cb WatchSource:0}: Error finding container 1d5d29deaccc74ece02177505f8cfe79188a7a935a63e1c557925b6f926347cb: Status 404 returned error can't find the container with id 1d5d29deaccc74ece02177505f8cfe79188a7a935a63e1c557925b6f926347cb
Jul 30 10:54:49.193840 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:49.193807    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb" event={"ID":"32a98e99-6467-42f3-9c62-e73f43e48a4b","Type":"ContainerStarted","Data":"1d5d29deaccc74ece02177505f8cfe79188a7a935a63e1c557925b6f926347cb"}
Jul 30 10:54:49.261443 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:49.261410    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de38d89b-a26e-4d4d-999f-00a1585e9484-tls-certs\") pod \"kube-auth-proxy-8674569cb6-mc94c\" (UID: \"de38d89b-a26e-4d4d-999f-00a1585e9484\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:49.263732 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:49.263708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de38d89b-a26e-4d4d-999f-00a1585e9484-tls-certs\") pod \"kube-auth-proxy-8674569cb6-mc94c\" (UID: \"de38d89b-a26e-4d4d-999f-00a1585e9484\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:49.524598 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:49.524563    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"
Jul 30 10:54:49.666793 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:49.666674    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8674569cb6-mc94c"]
Jul 30 10:54:49.669655 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:54:49.669631    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde38d89b_a26e_4d4d_999f_00a1585e9484.slice/crio-b68893ee9c08ba28bea3533b16292319d4871bbebcc9d8630ce68d88e6471d8c WatchSource:0}: Error finding container b68893ee9c08ba28bea3533b16292319d4871bbebcc9d8630ce68d88e6471d8c: Status 404 returned error can't find the container with id b68893ee9c08ba28bea3533b16292319d4871bbebcc9d8630ce68d88e6471d8c
Jul 30 10:54:50.199402 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:50.199363    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c" event={"ID":"de38d89b-a26e-4d4d-999f-00a1585e9484","Type":"ContainerStarted","Data":"b68893ee9c08ba28bea3533b16292319d4871bbebcc9d8630ce68d88e6471d8c"}
Jul 30 10:54:50.335862 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:50.335827    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:50.435627 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:50.435558    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:50.636034 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:50.635948    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:53.212896 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:53.212855    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c" event={"ID":"de38d89b-a26e-4d4d-999f-00a1585e9484","Type":"ContainerStarted","Data":"d9b0774c53cb512ce28b33cdae9ba3d247aeac2b704f1f4347be5450f87252ad"}
Jul 30 10:54:53.214209 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:53.214185    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb" event={"ID":"32a98e99-6467-42f3-9c62-e73f43e48a4b","Type":"ContainerStarted","Data":"ec5a53395230277fd20effc9b0c37e83d8ffebd3dc24c8c02ebce35153353a7a"}
Jul 30 10:54:53.214328 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:53.214313    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:54:53.253705 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:53.253664    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8674569cb6-mc94c" podStartSLOduration=2.67460131 podStartE2EDuration="5.253648652s" totalImagesPullingTime="2.579047342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:53.251445143 +0000 UTC m=+398.972329404" watchObservedRunningTime="2026-07-30 10:54:53.253648652 +0000 UTC m=+398.974532913"
Jul 30 10:54:53.279589 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:53.279529    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb" podStartSLOduration=2.835879152 podStartE2EDuration="5.279504049s" totalImagesPullingTime="2.443624897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:53.277819026 +0000 UTC m=+398.998703287" watchObservedRunningTime="2026-07-30 10:54:53.279504049 +0000 UTC m=+399.000388337"
Jul 30 10:54:56.632151 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:56.632076    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:56.742300 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:56.742266    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:57.028157 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:57.028121    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:57.176676 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:57.176643    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:57.368761 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:57.368681    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:59.140059 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:59.140027    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:59.260910 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:59.260875    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:59.342362 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:59.342325    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:54:59.443156 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:54:59.443125    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:55:00.494538 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:55:00.494506    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:55:00.569376 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:55:00.569339    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:55:00.665273 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:55:00.665237    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:55:03.241086 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:55:03.241050    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:55:04.220232 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:55:04.220201    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-866774c595-tv5rb"
Jul 30 10:55:26.277051 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:55:26.277014    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:55:33.785809 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:55:33.785770    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:55:49.634847 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:55:49.634762    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:55:49.728349 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:55:49.728317    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:08.851505 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:08.849476    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:11.460865 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:11.460796    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:11.837262 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:11.837171    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:12.263136 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:12.263087    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:13.383421 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:13.383386    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:13.480397 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:13.480361    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:13.583535 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:13.583505    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:15.505412 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:15.505360    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:15.619143 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:15.619092    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:15.836973 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:15.836849    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:17.440610 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:17.440570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:17.841117 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:17.841021    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:18.093297 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:18.093223    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:20.269926 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:20.269886    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:20.372870 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:20.372839    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:21.436279 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:21.436244    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:21.537023 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:21.536993    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:22.068736 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.068705    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:22.322519 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.322444    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"]
Jul 30 10:56:22.325295 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.325279    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:22.327926 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.327899    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 30 10:56:22.328217 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.328178    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qjxbw\""
Jul 30 10:56:22.328217 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.328200    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 30 10:56:22.328381 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.328244    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 30 10:56:22.328827 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.328804    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:22.329157 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.329141    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 30 10:56:22.333707 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.333684    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"]
Jul 30 10:56:22.446570 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.446532    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg9kk\" (UniqueName: \"kubernetes.io/projected/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-kube-api-access-mg9kk\") pod \"kuadrant-console-plugin-598f86468b-j6kf2\" (UID: \"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:22.446570 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.446573    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-j6kf2\" (UID: \"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:22.447089 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.446598    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-j6kf2\" (UID: \"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:22.547809 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.547778    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg9kk\" (UniqueName: \"kubernetes.io/projected/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-kube-api-access-mg9kk\") pod \"kuadrant-console-plugin-598f86468b-j6kf2\" (UID: \"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:22.548001 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.547819    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-j6kf2\" (UID: \"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:22.548001 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.547838    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-j6kf2\" (UID: \"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:22.548001 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:56:22.547942    2571 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 30 10:56:22.548187 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:56:22.548009    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-plugin-serving-cert podName:1f1ee5b9-b701-4c17-8eb0-6b1c55b22672 nodeName:}" failed. No retries permitted until 2026-07-30 10:56:23.047989213 +0000 UTC m=+488.768873472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-j6kf2" (UID: "1f1ee5b9-b701-4c17-8eb0-6b1c55b22672") : secret "plugin-serving-cert" not found
Jul 30 10:56:22.548471 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.548454    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-j6kf2\" (UID: \"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:22.556297 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.556275    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg9kk\" (UniqueName: \"kubernetes.io/projected/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-kube-api-access-mg9kk\") pod \"kuadrant-console-plugin-598f86468b-j6kf2\" (UID: \"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:22.719137 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:22.719086    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:23.052331 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:23.052247    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-j6kf2\" (UID: \"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:23.054521 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:23.054496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f1ee5b9-b701-4c17-8eb0-6b1c55b22672-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-j6kf2\" (UID: \"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:23.235067 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:23.235031    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"
Jul 30 10:56:23.361618 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:23.361590    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2"]
Jul 30 10:56:23.364333 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:56:23.364298    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f1ee5b9_b701_4c17_8eb0_6b1c55b22672.slice/crio-b4f12a11086c9cef49512bf0de623adc07e8af168392c1428da728b0f2253b3b WatchSource:0}: Error finding container b4f12a11086c9cef49512bf0de623adc07e8af168392c1428da728b0f2253b3b: Status 404 returned error can't find the container with id b4f12a11086c9cef49512bf0de623adc07e8af168392c1428da728b0f2253b3b
Jul 30 10:56:23.531696 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:23.531656    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2" event={"ID":"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672","Type":"ContainerStarted","Data":"b4f12a11086c9cef49512bf0de623adc07e8af168392c1428da728b0f2253b3b"}
Jul 30 10:56:23.830896 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:23.830859    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbfb7c986-b5dgg"]
Jul 30 10:56:23.948812 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:23.948761    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:24.050913 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:24.050879    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:24.107622 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:24.107501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:26.244278 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:26.244243    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:26.345482 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:26.345449    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:26.648342 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:26.648266    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:26.749427 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:26.749392    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:26.881189 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:26.881144    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:26.972647 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:26.972609    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:28.552293 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:28.552256    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2" event={"ID":"1f1ee5b9-b701-4c17-8eb0-6b1c55b22672","Type":"ContainerStarted","Data":"2f903725823784c05557fb36313e647fce41ab617018ba217cddb4a0a4e7d5df"}
Jul 30 10:56:28.569006 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:28.568959    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-j6kf2" podStartSLOduration=2.214955826 podStartE2EDuration="6.568944708s" totalImagesPullingTime="4.353988882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:56:28.568240403 +0000 UTC m=+494.289124665" watchObservedRunningTime="2026-07-30 10:56:28.568944708 +0000 UTC m=+494.289828969"
Jul 30 10:56:33.064878 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:33.064827    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:33.169370 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:33.169336    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:37.862805 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:37.862763    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:37.957092 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:37.957057    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:41.501922 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:41.501874    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:41.601384 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:41.601331    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:44.958822 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:44.958788    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:45.039213 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:45.039177    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:45.105821 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:45.105759    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:48.849288 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:48.849248    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fbfb7c986-b5dgg" podUID="6a16f59a-a67e-4c0d-a7b2-f31149946a7c" containerName="console" containerID="cri-o://9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4" gracePeriod=15
Jul 30 10:56:49.084844 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.084820    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:56:49.248317 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.248279    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-config\") pod \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") "
Jul 30 10:56:49.248508 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.248342    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65xp2\" (UniqueName: \"kubernetes.io/projected/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-kube-api-access-65xp2\") pod \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") "
Jul 30 10:56:49.248508 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.248380    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-trusted-ca-bundle\") pod \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") "
Jul 30 10:56:49.248508 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.248431    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-oauth-serving-cert\") pod \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") "
Jul 30 10:56:49.248508 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.248468    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-oauth-config\") pod \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") "
Jul 30 10:56:49.248508 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.248510    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-serving-cert\") pod \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") "
Jul 30 10:56:49.248769 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.248534    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-service-ca\") pod \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\" (UID: \"6a16f59a-a67e-4c0d-a7b2-f31149946a7c\") "
Jul 30 10:56:49.248912 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.248881    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-config" (OuterVolumeSpecName: "console-config") pod "6a16f59a-a67e-4c0d-a7b2-f31149946a7c" (UID: "6a16f59a-a67e-4c0d-a7b2-f31149946a7c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:56:49.248912 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.248899    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6a16f59a-a67e-4c0d-a7b2-f31149946a7c" (UID: "6a16f59a-a67e-4c0d-a7b2-f31149946a7c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:56:49.249061 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.248978    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6a16f59a-a67e-4c0d-a7b2-f31149946a7c" (UID: "6a16f59a-a67e-4c0d-a7b2-f31149946a7c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:56:49.249061 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.249009    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-service-ca" (OuterVolumeSpecName: "service-ca") pod "6a16f59a-a67e-4c0d-a7b2-f31149946a7c" (UID: "6a16f59a-a67e-4c0d-a7b2-f31149946a7c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 10:56:49.250567 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.250540    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-kube-api-access-65xp2" (OuterVolumeSpecName: "kube-api-access-65xp2") pod "6a16f59a-a67e-4c0d-a7b2-f31149946a7c" (UID: "6a16f59a-a67e-4c0d-a7b2-f31149946a7c"). InnerVolumeSpecName "kube-api-access-65xp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 10:56:49.250670 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.250628    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6a16f59a-a67e-4c0d-a7b2-f31149946a7c" (UID: "6a16f59a-a67e-4c0d-a7b2-f31149946a7c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 10:56:49.250732 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.250702    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6a16f59a-a67e-4c0d-a7b2-f31149946a7c" (UID: "6a16f59a-a67e-4c0d-a7b2-f31149946a7c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 10:56:49.349805 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.349776    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-trusted-ca-bundle\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:56:49.349805 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.349800    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-oauth-serving-cert\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:56:49.349805 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.349809    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-oauth-config\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:56:49.350002 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.349818    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-serving-cert\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:56:49.350002 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.349827    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:56:49.350002 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.349835    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-console-config\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:56:49.350002 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.349842    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65xp2\" (UniqueName: \"kubernetes.io/projected/6a16f59a-a67e-4c0d-a7b2-f31149946a7c-kube-api-access-65xp2\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:56:49.623903 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.623802    2571 generic.go:356] "Generic (PLEG): container finished" podID="6a16f59a-a67e-4c0d-a7b2-f31149946a7c" containerID="9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4" exitCode=0
Jul 30 10:56:49.623903 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.623864    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbfb7c986-b5dgg"
Jul 30 10:56:49.623903 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.623876    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbfb7c986-b5dgg" event={"ID":"6a16f59a-a67e-4c0d-a7b2-f31149946a7c","Type":"ContainerDied","Data":"9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4"}
Jul 30 10:56:49.624206 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.623951    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbfb7c986-b5dgg" event={"ID":"6a16f59a-a67e-4c0d-a7b2-f31149946a7c","Type":"ContainerDied","Data":"92dee2c043f376cdc2af97004dca566f667f47697e9bb34f0f81aa61a75635ca"}
Jul 30 10:56:49.624206 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.623971    2571 scope.go:117] "RemoveContainer" containerID="9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4"
Jul 30 10:56:49.632468 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.632451    2571 scope.go:117] "RemoveContainer" containerID="9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4"
Jul 30 10:56:49.632698 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:56:49.632681    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4\": container with ID starting with 9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4 not found: ID does not exist" containerID="9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4"
Jul 30 10:56:49.632733 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.632708    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4"} err="failed to get container status \"9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4\": rpc error: code = NotFound desc = could not find container \"9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4\": container with ID starting with 9a43f88987a777336f24918f5ff578ef0c04391919638094b7002b09f84dfbb4 not found: ID does not exist"
Jul 30 10:56:49.647523 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.647500    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbfb7c986-b5dgg"]
Jul 30 10:56:49.653513 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:49.653491    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fbfb7c986-b5dgg"]
Jul 30 10:56:50.854897 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:50.854862    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a16f59a-a67e-4c0d-a7b2-f31149946a7c" path="/var/lib/kubelet/pods/6a16f59a-a67e-4c0d-a7b2-f31149946a7c/volumes"
Jul 30 10:56:52.006345 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:52.006249    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:52.107544 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:52.107509    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:56.800859 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:56.800821    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:56.900040 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:56.899980    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:59.399451 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:59.399417    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:56:59.500340 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:56:59.500304    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:02.484979 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:02.484942    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:02.581482 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:02.581450    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:04.810841 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:04.810805    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:04.911239 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:04.911210    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:05.069595 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.069505    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:05.179780 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.179744    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:05.276603 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.276554    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:05.541200 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.541168    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 10:57:05.541424 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.541413    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a16f59a-a67e-4c0d-a7b2-f31149946a7c" containerName="console"
Jul 30 10:57:05.541468 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.541426    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a16f59a-a67e-4c0d-a7b2-f31149946a7c" containerName="console"
Jul 30 10:57:05.541501 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.541475    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a16f59a-a67e-4c0d-a7b2-f31149946a7c" containerName="console"
Jul 30 10:57:05.560897 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.560873    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 10:57:05.561045 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.560965    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"
Jul 30 10:57:05.563410 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.563384    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 30 10:57:05.565757 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.565738    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 10:57:05.618536 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.618505    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:05.660867 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.660841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc0d75c4-8753-48f0-8104-9bc0e2304a55-config-file\") pod \"limitador-limitador-85d9db4d5d-sthk2\" (UID: \"fc0d75c4-8753-48f0-8104-9bc0e2304a55\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"
Jul 30 10:57:05.660966 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.660883    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6p8\" (UniqueName: \"kubernetes.io/projected/fc0d75c4-8753-48f0-8104-9bc0e2304a55-kube-api-access-xc6p8\") pod \"limitador-limitador-85d9db4d5d-sthk2\" (UID: \"fc0d75c4-8753-48f0-8104-9bc0e2304a55\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"
Jul 30 10:57:05.689633 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.689610    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:05.761514 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.761489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc0d75c4-8753-48f0-8104-9bc0e2304a55-config-file\") pod \"limitador-limitador-85d9db4d5d-sthk2\" (UID: \"fc0d75c4-8753-48f0-8104-9bc0e2304a55\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"
Jul 30 10:57:05.761645 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.761537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6p8\" (UniqueName: \"kubernetes.io/projected/fc0d75c4-8753-48f0-8104-9bc0e2304a55-kube-api-access-xc6p8\") pod \"limitador-limitador-85d9db4d5d-sthk2\" (UID: \"fc0d75c4-8753-48f0-8104-9bc0e2304a55\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"
Jul 30 10:57:05.762186 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.762167    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc0d75c4-8753-48f0-8104-9bc0e2304a55-config-file\") pod \"limitador-limitador-85d9db4d5d-sthk2\" (UID: \"fc0d75c4-8753-48f0-8104-9bc0e2304a55\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"
Jul 30 10:57:05.775804 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.775776    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6p8\" (UniqueName: \"kubernetes.io/projected/fc0d75c4-8753-48f0-8104-9bc0e2304a55-kube-api-access-xc6p8\") pod \"limitador-limitador-85d9db4d5d-sthk2\" (UID: \"fc0d75c4-8753-48f0-8104-9bc0e2304a55\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"
Jul 30 10:57:05.870561 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.870491    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"
Jul 30 10:57:05.897867 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.897841    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-z52r5"]
Jul 30 10:57:05.900933 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.900915    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-z52r5"
Jul 30 10:57:05.903608 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.903590    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x9m5r\""
Jul 30 10:57:05.906302 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.906277    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-z52r5"]
Jul 30 10:57:05.926857 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.926823    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:05.963572 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:05.963542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lklv\" (UniqueName: \"kubernetes.io/projected/17a2635d-02c0-4b7a-a983-94f3643de3c7-kube-api-access-6lklv\") pod \"authorino-b54df94b4-z52r5\" (UID: \"17a2635d-02c0-4b7a-a983-94f3643de3c7\") " pod="kuadrant-system/authorino-b54df94b4-z52r5"
Jul 30 10:57:06.001313 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:06.001291    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 10:57:06.003787 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:57:06.003752    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0d75c4_8753_48f0_8104_9bc0e2304a55.slice/crio-9d80d0201b0429c52c54af38b9b9eb2a137885615a127d66589e8ea73bb1c133 WatchSource:0}: Error finding container 9d80d0201b0429c52c54af38b9b9eb2a137885615a127d66589e8ea73bb1c133: Status 404 returned error can't find the container with id 9d80d0201b0429c52c54af38b9b9eb2a137885615a127d66589e8ea73bb1c133
Jul 30 10:57:06.004494 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:06.004476    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:06.064496 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:06.064466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lklv\" (UniqueName: \"kubernetes.io/projected/17a2635d-02c0-4b7a-a983-94f3643de3c7-kube-api-access-6lklv\") pod \"authorino-b54df94b4-z52r5\" (UID: \"17a2635d-02c0-4b7a-a983-94f3643de3c7\") " pod="kuadrant-system/authorino-b54df94b4-z52r5"
Jul 30 10:57:06.072562 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:06.072534    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lklv\" (UniqueName: \"kubernetes.io/projected/17a2635d-02c0-4b7a-a983-94f3643de3c7-kube-api-access-6lklv\") pod \"authorino-b54df94b4-z52r5\" (UID: \"17a2635d-02c0-4b7a-a983-94f3643de3c7\") " pod="kuadrant-system/authorino-b54df94b4-z52r5"
Jul 30 10:57:06.105404 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:06.105378    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:06.214184 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:06.214158    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-z52r5"
Jul 30 10:57:06.336394 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:06.336336    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-z52r5"]
Jul 30 10:57:06.688063 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:06.688026    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-z52r5" event={"ID":"17a2635d-02c0-4b7a-a983-94f3643de3c7","Type":"ContainerStarted","Data":"74ccfa6cdb33eb0e6a7fd0ca583dcc6983b3aa966ac804a29104070a4f5e703f"}
Jul 30 10:57:06.689836 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:06.689800    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2" event={"ID":"fc0d75c4-8753-48f0-8104-9bc0e2304a55","Type":"ContainerStarted","Data":"9d80d0201b0429c52c54af38b9b9eb2a137885615a127d66589e8ea73bb1c133"}
Jul 30 10:57:08.700599 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:08.700558    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2" event={"ID":"fc0d75c4-8753-48f0-8104-9bc0e2304a55","Type":"ContainerStarted","Data":"773366e97acded7301edf38cd9d241e42e38638eb2fe2db0404804a96605a7b1"}
Jul 30 10:57:08.700965 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:08.700691    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"
Jul 30 10:57:08.718067 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:08.718018    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2" podStartSLOduration=1.605732723 podStartE2EDuration="3.71800394s" totalImagesPullingTime="2.112271217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:57:08.717192922 +0000 UTC m=+534.438077182" watchObservedRunningTime="2026-07-30 10:57:08.71800394 +0000 UTC m=+534.438888201"
Jul 30 10:57:09.263060 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:09.263026    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-z52r5"]
Jul 30 10:57:12.252823 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:12.252779    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:12.352151 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:12.352116    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:17.843994 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:17.843958    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:17.943778 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:17.943734    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:19.705851 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:19.705607    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"
Jul 30 10:57:20.511033 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:20.510949    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:20.574872 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:20.574839    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:20.745274 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:20.745243    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-z52r5" event={"ID":"17a2635d-02c0-4b7a-a983-94f3643de3c7","Type":"ContainerStarted","Data":"c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68"}
Jul 30 10:57:20.745626 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:20.745310    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-z52r5" podUID="17a2635d-02c0-4b7a-a983-94f3643de3c7" containerName="authorino" containerID="cri-o://c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68" gracePeriod=30
Jul 30 10:57:20.762261 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:20.762190    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-z52r5" podStartSLOduration=1.8497220570000001 podStartE2EDuration="15.762175511s" totalImagesPullingTime="13.912453454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:57:20.76054635 +0000 UTC m=+546.481430610" watchObservedRunningTime="2026-07-30 10:57:20.762175511 +0000 UTC m=+546.483059772"
Jul 30 10:57:20.983424 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:20.983399    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-z52r5"
Jul 30 10:57:21.079502 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.079418    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lklv\" (UniqueName: \"kubernetes.io/projected/17a2635d-02c0-4b7a-a983-94f3643de3c7-kube-api-access-6lklv\") pod \"17a2635d-02c0-4b7a-a983-94f3643de3c7\" (UID: \"17a2635d-02c0-4b7a-a983-94f3643de3c7\") "
Jul 30 10:57:21.081633 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.081603    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a2635d-02c0-4b7a-a983-94f3643de3c7-kube-api-access-6lklv" (OuterVolumeSpecName: "kube-api-access-6lklv") pod "17a2635d-02c0-4b7a-a983-94f3643de3c7" (UID: "17a2635d-02c0-4b7a-a983-94f3643de3c7"). InnerVolumeSpecName "kube-api-access-6lklv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 10:57:21.089822 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.089798    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:21.185172 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.180491    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lklv\" (UniqueName: \"kubernetes.io/projected/17a2635d-02c0-4b7a-a983-94f3643de3c7-kube-api-access-6lklv\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 10:57:21.185340 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.185255    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:21.750644 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.750610    2571 generic.go:356] "Generic (PLEG): container finished" podID="17a2635d-02c0-4b7a-a983-94f3643de3c7" containerID="c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68" exitCode=0
Jul 30 10:57:21.751060 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.750670    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-z52r5" event={"ID":"17a2635d-02c0-4b7a-a983-94f3643de3c7","Type":"ContainerDied","Data":"c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68"}
Jul 30 10:57:21.751060 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.750674    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-z52r5"
Jul 30 10:57:21.751060 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.750701    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-z52r5" event={"ID":"17a2635d-02c0-4b7a-a983-94f3643de3c7","Type":"ContainerDied","Data":"74ccfa6cdb33eb0e6a7fd0ca583dcc6983b3aa966ac804a29104070a4f5e703f"}
Jul 30 10:57:21.751060 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.750718    2571 scope.go:117] "RemoveContainer" containerID="c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68"
Jul 30 10:57:21.758825 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.758807    2571 scope.go:117] "RemoveContainer" containerID="c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68"
Jul 30 10:57:21.759117 ip-10-0-128-243 kubenswrapper[2571]: E0730 10:57:21.759082    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68\": container with ID starting with c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68 not found: ID does not exist" containerID="c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68"
Jul 30 10:57:21.759202 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.759122    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68"} err="failed to get container status \"c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68\": rpc error: code = NotFound desc = could not find container \"c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68\": container with ID starting with c6b636bee493242aa42a062827206c39d7928dc7ee2ba3367278f35bcdfbfe68 not found: ID does not exist"
Jul 30 10:57:21.770042 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.770017    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-z52r5"]
Jul 30 10:57:21.771640 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:21.771619    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-z52r5"]
Jul 30 10:57:22.854223 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:22.854188    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a2635d-02c0-4b7a-a983-94f3643de3c7" path="/var/lib/kubelet/pods/17a2635d-02c0-4b7a-a983-94f3643de3c7/volumes"
Jul 30 10:57:23.464192 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:23.464158    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:33.676342 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:33.676301    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:33.777734 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:33.777699    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:34.186663 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.186625    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76575f49f8-8qnpl"]
Jul 30 10:57:34.186908 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.186895    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a2635d-02c0-4b7a-a983-94f3643de3c7" containerName="authorino"
Jul 30 10:57:34.186960 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.186910    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a2635d-02c0-4b7a-a983-94f3643de3c7" containerName="authorino"
Jul 30 10:57:34.186960 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.186952    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a2635d-02c0-4b7a-a983-94f3643de3c7" containerName="authorino"
Jul 30 10:57:34.191180 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.191163    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76575f49f8-8qnpl"
Jul 30 10:57:34.195196 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.195176    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 30 10:57:34.195310 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.195215    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x9m5r\""
Jul 30 10:57:34.199193 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.199173    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76575f49f8-8qnpl"]
Jul 30 10:57:34.284878 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.284844    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/626f2d2f-bb8b-4d68-91b3-7f51219df362-tls-cert\") pod \"authorino-76575f49f8-8qnpl\" (UID: \"626f2d2f-bb8b-4d68-91b3-7f51219df362\") " pod="kuadrant-system/authorino-76575f49f8-8qnpl"
Jul 30 10:57:34.285035 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.284950    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w82sf\" (UniqueName: \"kubernetes.io/projected/626f2d2f-bb8b-4d68-91b3-7f51219df362-kube-api-access-w82sf\") pod \"authorino-76575f49f8-8qnpl\" (UID: \"626f2d2f-bb8b-4d68-91b3-7f51219df362\") " pod="kuadrant-system/authorino-76575f49f8-8qnpl"
Jul 30 10:57:34.386016 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.385985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w82sf\" (UniqueName: \"kubernetes.io/projected/626f2d2f-bb8b-4d68-91b3-7f51219df362-kube-api-access-w82sf\") pod \"authorino-76575f49f8-8qnpl\" (UID: \"626f2d2f-bb8b-4d68-91b3-7f51219df362\") " pod="kuadrant-system/authorino-76575f49f8-8qnpl"
Jul 30 10:57:34.386171 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.386025    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/626f2d2f-bb8b-4d68-91b3-7f51219df362-tls-cert\") pod \"authorino-76575f49f8-8qnpl\" (UID: \"626f2d2f-bb8b-4d68-91b3-7f51219df362\") " pod="kuadrant-system/authorino-76575f49f8-8qnpl"
Jul 30 10:57:34.388342 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.388325    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/626f2d2f-bb8b-4d68-91b3-7f51219df362-tls-cert\") pod \"authorino-76575f49f8-8qnpl\" (UID: \"626f2d2f-bb8b-4d68-91b3-7f51219df362\") " pod="kuadrant-system/authorino-76575f49f8-8qnpl"
Jul 30 10:57:34.397343 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.397315    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w82sf\" (UniqueName: \"kubernetes.io/projected/626f2d2f-bb8b-4d68-91b3-7f51219df362-kube-api-access-w82sf\") pod \"authorino-76575f49f8-8qnpl\" (UID: \"626f2d2f-bb8b-4d68-91b3-7f51219df362\") " pod="kuadrant-system/authorino-76575f49f8-8qnpl"
Jul 30 10:57:34.500083 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.499996    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76575f49f8-8qnpl"
Jul 30 10:57:34.623407 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.623379    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76575f49f8-8qnpl"]
Jul 30 10:57:34.629034 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:57:34.629007    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod626f2d2f_bb8b_4d68_91b3_7f51219df362.slice/crio-eb391e2ffcd8687ec658fe382c30e503ce748ad0f5699495e5d4031d5addaf77 WatchSource:0}: Error finding container eb391e2ffcd8687ec658fe382c30e503ce748ad0f5699495e5d4031d5addaf77: Status 404 returned error can't find the container with id eb391e2ffcd8687ec658fe382c30e503ce748ad0f5699495e5d4031d5addaf77
Jul 30 10:57:34.804772 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:34.804686    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76575f49f8-8qnpl" event={"ID":"626f2d2f-bb8b-4d68-91b3-7f51219df362","Type":"ContainerStarted","Data":"eb391e2ffcd8687ec658fe382c30e503ce748ad0f5699495e5d4031d5addaf77"}
Jul 30 10:57:35.809867 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:35.809835    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76575f49f8-8qnpl" event={"ID":"626f2d2f-bb8b-4d68-91b3-7f51219df362","Type":"ContainerStarted","Data":"6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c"}
Jul 30 10:57:35.828834 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:35.828786    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76575f49f8-8qnpl" podStartSLOduration=1.1408334230000001 podStartE2EDuration="1.828770649s" totalImagesPullingTime="687.937226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:57:35.827603809 +0000 UTC m=+561.548488071" watchObservedRunningTime="2026-07-30 10:57:35.828770649 +0000 UTC m=+561.549654910"
Jul 30 10:57:40.557564 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:40.557528    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:40.661644 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:40.661613    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:40.722995 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:40.722948    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:43.887187 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:43.887152    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:57:43.983663 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:57:43.983631    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:15.553653 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:15.553617    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:15.656327 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:15.656293    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:16.961821 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:16.961790    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:17.071759 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:17.071727    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:17.165754 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:17.165720    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:17.992334 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:17.992301    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 10:58:18.161501 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:18.161466    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:18.260349 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:18.260270    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:27.249115 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:27.249062    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:28.505943 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:28.505909    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:37.712026 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:37.711947    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:58:38.125716 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:38.125634    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 10:58:54.705849 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:58:54.705808    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:59:00.125728 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:00.125682    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:59:00.896837 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:00.896800    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 10:59:05.209839 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:05.209799    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:59:10.317079 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:10.317041    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:59:10.423989 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:10.423955    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:59:21.614046 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:21.614014    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:59:22.492028 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:22.491982    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 10:59:37.819009 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:37.818973    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:59:38.593141 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:38.593091    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 10:59:47.132460 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:47.132421    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:59:49.582260 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:49.582226    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 10:59:52.886063 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.886033    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"]
Jul 30 10:59:52.889482 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.889462    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:52.893540 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.893505    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f52tt\""
Jul 30 10:59:52.893661 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.893506    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 30 10:59:52.893661 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.893514    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 30 10:59:52.893661 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.893575    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 30 10:59:52.899739 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.899722    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"]
Jul 30 10:59:52.930222 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.930198    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 10:59:52.989222 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.989192    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:52.989222 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.989221    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d4ceca-d270-404e-a301-a0a840cb5ef7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:52.989358 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.989246    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkkhk\" (UniqueName: \"kubernetes.io/projected/83d4ceca-d270-404e-a301-a0a840cb5ef7-kube-api-access-rkkhk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:52.989358 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.989314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:52.989358 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.989349    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:52.989507 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:52.989368    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.089714 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.089686    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.089828 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.089723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.089828 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.089740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.089828 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.089785    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.089828 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.089817    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d4ceca-d270-404e-a301-a0a840cb5ef7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.090037 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.089852    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkkhk\" (UniqueName: \"kubernetes.io/projected/83d4ceca-d270-404e-a301-a0a840cb5ef7-kube-api-access-rkkhk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.090183 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.090162    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.090249 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.090208    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.090249 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.090237    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.092034 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.092015    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d4ceca-d270-404e-a301-a0a840cb5ef7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.092317 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.092301    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d4ceca-d270-404e-a301-a0a840cb5ef7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.100042 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.100019    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkkhk\" (UniqueName: \"kubernetes.io/projected/83d4ceca-d270-404e-a301-a0a840cb5ef7-kube-api-access-rkkhk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2\" (UID: \"83d4ceca-d270-404e-a301-a0a840cb5ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.200047 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.200024    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 10:59:53.328751 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.328720    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"]
Jul 30 10:59:53.331288 ip-10-0-128-243 kubenswrapper[2571]: W0730 10:59:53.331249    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83d4ceca_d270_404e_a301_a0a840cb5ef7.slice/crio-4bc0551cddbfa920cedf6e9e29e26f9faffeef5c8612c4b63067c57d236fab47 WatchSource:0}: Error finding container 4bc0551cddbfa920cedf6e9e29e26f9faffeef5c8612c4b63067c57d236fab47: Status 404 returned error can't find the container with id 4bc0551cddbfa920cedf6e9e29e26f9faffeef5c8612c4b63067c57d236fab47
Jul 30 10:59:53.332524 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:53.332508    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 10:59:54.293385 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:54.293351    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2" event={"ID":"83d4ceca-d270-404e-a301-a0a840cb5ef7","Type":"ContainerStarted","Data":"4bc0551cddbfa920cedf6e9e29e26f9faffeef5c8612c4b63067c57d236fab47"}
Jul 30 10:59:55.299629 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:55.299590    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2" event={"ID":"83d4ceca-d270-404e-a301-a0a840cb5ef7","Type":"ContainerStarted","Data":"2a8d317cc700ff79c46c2a338a962d3fbb1eeb1d25f4aaed94d6e81d9e3e6591"}
Jul 30 10:59:55.319418 ip-10-0-128-243 kubenswrapper[2571]: I0730 10:59:55.319366    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2" podStartSLOduration=2.324048452 podStartE2EDuration="3.319350796s" totalImagesPullingTime="995.302344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:59:55.318377387 +0000 UTC m=+701.039261648" watchObservedRunningTime="2026-07-30 10:59:55.319350796 +0000 UTC m=+701.040235057"
Jul 30 11:00:00.889760 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:00:00.889719    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:00:03.200998 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:00:03.200959    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 11:00:03.200998 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:00:03.201007    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 11:00:03.213512 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:00:03.213482    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 11:00:03.348264 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:00:03.348232    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2"
Jul 30 11:00:27.982382 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:00:27.982298    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:01:04.470112 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.470077    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f6748df59-rb6zl"]
Jul 30 11:01:04.473302 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.473283    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f6748df59-rb6zl"
Jul 30 11:01:04.481739 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.481712    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f6748df59-rb6zl"]
Jul 30 11:01:04.626034 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.625996    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fe25eefe-9d08-4719-a944-7859dcf49aff-tls-cert\") pod \"authorino-7f6748df59-rb6zl\" (UID: \"fe25eefe-9d08-4719-a944-7859dcf49aff\") " pod="kuadrant-system/authorino-7f6748df59-rb6zl"
Jul 30 11:01:04.626034 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.626035    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngmtb\" (UniqueName: \"kubernetes.io/projected/fe25eefe-9d08-4719-a944-7859dcf49aff-kube-api-access-ngmtb\") pod \"authorino-7f6748df59-rb6zl\" (UID: \"fe25eefe-9d08-4719-a944-7859dcf49aff\") " pod="kuadrant-system/authorino-7f6748df59-rb6zl"
Jul 30 11:01:04.727081 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.727003    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fe25eefe-9d08-4719-a944-7859dcf49aff-tls-cert\") pod \"authorino-7f6748df59-rb6zl\" (UID: \"fe25eefe-9d08-4719-a944-7859dcf49aff\") " pod="kuadrant-system/authorino-7f6748df59-rb6zl"
Jul 30 11:01:04.727081 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.727036    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngmtb\" (UniqueName: \"kubernetes.io/projected/fe25eefe-9d08-4719-a944-7859dcf49aff-kube-api-access-ngmtb\") pod \"authorino-7f6748df59-rb6zl\" (UID: \"fe25eefe-9d08-4719-a944-7859dcf49aff\") " pod="kuadrant-system/authorino-7f6748df59-rb6zl"
Jul 30 11:01:04.729423 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.729392    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fe25eefe-9d08-4719-a944-7859dcf49aff-tls-cert\") pod \"authorino-7f6748df59-rb6zl\" (UID: \"fe25eefe-9d08-4719-a944-7859dcf49aff\") " pod="kuadrant-system/authorino-7f6748df59-rb6zl"
Jul 30 11:01:04.736070 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.736050    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngmtb\" (UniqueName: \"kubernetes.io/projected/fe25eefe-9d08-4719-a944-7859dcf49aff-kube-api-access-ngmtb\") pod \"authorino-7f6748df59-rb6zl\" (UID: \"fe25eefe-9d08-4719-a944-7859dcf49aff\") " pod="kuadrant-system/authorino-7f6748df59-rb6zl"
Jul 30 11:01:04.782866 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.782842    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f6748df59-rb6zl"
Jul 30 11:01:04.917974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:04.915678    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f6748df59-rb6zl"]
Jul 30 11:01:04.920373 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:01:04.920336    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe25eefe_9d08_4719_a944_7859dcf49aff.slice/crio-cb6df8cf627426756adb9a67a97473410ac2543f2cb6aae1c979a27747c28595 WatchSource:0}: Error finding container cb6df8cf627426756adb9a67a97473410ac2543f2cb6aae1c979a27747c28595: Status 404 returned error can't find the container with id cb6df8cf627426756adb9a67a97473410ac2543f2cb6aae1c979a27747c28595
Jul 30 11:01:05.578263 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:05.578226    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f6748df59-rb6zl" event={"ID":"fe25eefe-9d08-4719-a944-7859dcf49aff","Type":"ContainerStarted","Data":"cb6df8cf627426756adb9a67a97473410ac2543f2cb6aae1c979a27747c28595"}
Jul 30 11:01:06.584823 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:06.584781    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f6748df59-rb6zl" event={"ID":"fe25eefe-9d08-4719-a944-7859dcf49aff","Type":"ContainerStarted","Data":"70c01b3373c423577d8c45fc0515792d7b311dcadb6a665c8a1e4031e6ee9db4"}
Jul 30 11:01:06.602554 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:06.602495    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f6748df59-rb6zl" podStartSLOduration=1.91364285 podStartE2EDuration="2.602475272s" totalImagesPullingTime="688.832422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:01:06.602086528 +0000 UTC m=+772.322970798" watchObservedRunningTime="2026-07-30 11:01:06.602475272 +0000 UTC m=+772.323359533"
Jul 30 11:01:06.638000 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:06.637963    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76575f49f8-8qnpl"]
Jul 30 11:01:06.638221 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:06.638197    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-76575f49f8-8qnpl" podUID="626f2d2f-bb8b-4d68-91b3-7f51219df362" containerName="authorino" containerID="cri-o://6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c" gracePeriod=30
Jul 30 11:01:06.900990 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:06.900958    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76575f49f8-8qnpl"
Jul 30 11:01:07.049143 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.049082    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/626f2d2f-bb8b-4d68-91b3-7f51219df362-tls-cert\") pod \"626f2d2f-bb8b-4d68-91b3-7f51219df362\" (UID: \"626f2d2f-bb8b-4d68-91b3-7f51219df362\") "
Jul 30 11:01:07.049332 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.049196    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w82sf\" (UniqueName: \"kubernetes.io/projected/626f2d2f-bb8b-4d68-91b3-7f51219df362-kube-api-access-w82sf\") pod \"626f2d2f-bb8b-4d68-91b3-7f51219df362\" (UID: \"626f2d2f-bb8b-4d68-91b3-7f51219df362\") "
Jul 30 11:01:07.051233 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.051198    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626f2d2f-bb8b-4d68-91b3-7f51219df362-kube-api-access-w82sf" (OuterVolumeSpecName: "kube-api-access-w82sf") pod "626f2d2f-bb8b-4d68-91b3-7f51219df362" (UID: "626f2d2f-bb8b-4d68-91b3-7f51219df362"). InnerVolumeSpecName "kube-api-access-w82sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:01:07.057932 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.057909    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626f2d2f-bb8b-4d68-91b3-7f51219df362-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "626f2d2f-bb8b-4d68-91b3-7f51219df362" (UID: "626f2d2f-bb8b-4d68-91b3-7f51219df362"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:01:07.141114 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.141016    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:01:07.150158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.150136    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w82sf\" (UniqueName: \"kubernetes.io/projected/626f2d2f-bb8b-4d68-91b3-7f51219df362-kube-api-access-w82sf\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:01:07.150158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.150159    2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/626f2d2f-bb8b-4d68-91b3-7f51219df362-tls-cert\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:01:07.247426 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.247395    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:01:07.591501 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.591467    2571 generic.go:356] "Generic (PLEG): container finished" podID="626f2d2f-bb8b-4d68-91b3-7f51219df362" containerID="6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c" exitCode=0
Jul 30 11:01:07.591906 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.591523    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76575f49f8-8qnpl"
Jul 30 11:01:07.591906 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.591515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76575f49f8-8qnpl" event={"ID":"626f2d2f-bb8b-4d68-91b3-7f51219df362","Type":"ContainerDied","Data":"6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c"}
Jul 30 11:01:07.591906 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.591649    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76575f49f8-8qnpl" event={"ID":"626f2d2f-bb8b-4d68-91b3-7f51219df362","Type":"ContainerDied","Data":"eb391e2ffcd8687ec658fe382c30e503ce748ad0f5699495e5d4031d5addaf77"}
Jul 30 11:01:07.591906 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.591676    2571 scope.go:117] "RemoveContainer" containerID="6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c"
Jul 30 11:01:07.600985 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.600954    2571 scope.go:117] "RemoveContainer" containerID="6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c"
Jul 30 11:01:07.601463 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:01:07.601433    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c\": container with ID starting with 6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c not found: ID does not exist" containerID="6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c"
Jul 30 11:01:07.601574 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.601473    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c"} err="failed to get container status \"6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c\": rpc error: code = NotFound desc = could not find container \"6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c\": container with ID starting with 6a109c053b057f87946aada25b217116e673e9cb0d6930aa199314689dd5510c not found: ID does not exist"
Jul 30 11:01:07.617518 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.617491    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76575f49f8-8qnpl"]
Jul 30 11:01:07.619020 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:07.618993    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76575f49f8-8qnpl"]
Jul 30 11:01:08.232732 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:08.232697    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:01:08.339459 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:08.339423    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:01:08.432509 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:08.432468    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:01:08.854781 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:08.854751    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="626f2d2f-bb8b-4d68-91b3-7f51219df362" path="/var/lib/kubelet/pods/626f2d2f-bb8b-4d68-91b3-7f51219df362/volumes"
Jul 30 11:01:35.610954 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:35.610921    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:01:45.591245 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:45.591166    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:01:58.603821 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:01:58.603782    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:02:02.529502 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:02:02.529467    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:03:12.870051 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:03:12.869971    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:03:14.130410 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:03:14.130374    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:03:22.638409 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:03:22.638375    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:03:36.789753 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:03:36.789716    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:03:46.594078 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:03:46.594034    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:04:14.980399 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:14.980364    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:16.313919 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:16.313888    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:21.515914 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:21.515879    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:21.633912 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:21.633867    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:21.758164 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:21.758123    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:21.894087 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:21.894006    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:28.059491 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:28.059456    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:28.215874 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:28.215843    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:33.944046 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:33.944012    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:36.352380 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:36.352335    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:37.293484 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:37.293447    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:37.380623 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:37.380585    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:04:44.496618 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:44.496575    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:04:51.099111 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:04:51.099074    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:05:06.694021 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:05:06.693984    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:05:12.892728 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:05:12.892690    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:05:22.506720 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:05:22.506686    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:05:30.390039 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:05:30.390000    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:05:42.590145 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:05:42.590090    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:05:48.187265 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:05:48.187232    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:05:56.090132 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:05:56.090077    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:06:04.886696 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:06:04.886663    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:06:17.696449 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:06:17.696373    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:06:28.391663 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:06:28.391626    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:06:51.789576 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:06:51.789540    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:07:01.070596 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:07:01.070553    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:07:01.990764 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:07:01.990733    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:07:07.290146 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:07:07.290093    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:07:15.087746 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:07:15.087712    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:07:23.191865 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:07:23.191827    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:07:27.589454 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:07:27.589417    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:07:33.985788 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:07:33.985747    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:07:40.590199 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:07:40.590118    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:08:03.493369 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:08:03.493332    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:08:12.991021 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:08:12.990985    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:08:22.232458 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:08:22.232412    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:08:32.892826 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:08:32.892790    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:08:38.101347 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:08:38.101312    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:08:48.781510 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:08:48.781474    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:10:39.583406 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:10:39.583321    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:11:05.291308 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:11:05.291272    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:11:12.381555 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:11:12.381517    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:11:34.283032 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:11:34.282991    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:12:13.586637 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:12:13.586543    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:12:20.084539 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:12:20.084504    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:12:29.384889 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:12:29.384853    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:12:34.393212 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:12:34.393177    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:12:45.691204 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:12:45.691166    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:12:55.491454 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:12:55.491418    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:13:00.395834 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:13:00.395799    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:13:05.391039 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:13:05.390999    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:13:06.135775 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:13:06.135739    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:13:16.191398 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:13:16.191359    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:13:22.494971 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:13:22.494932    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:13:32.390853 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:13:32.390816    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:13:46.316802 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:13:46.316759    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:13:57.384722 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:13:57.384690    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:14:02.289241 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:02.289202    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:14:05.254807 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.254768    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"]
Jul 30 11:14:05.255279 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.255073    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626f2d2f-bb8b-4d68-91b3-7f51219df362" containerName="authorino"
Jul 30 11:14:05.255279 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.255087    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="626f2d2f-bb8b-4d68-91b3-7f51219df362" containerName="authorino"
Jul 30 11:14:05.255279 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.255164    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="626f2d2f-bb8b-4d68-91b3-7f51219df362" containerName="authorino"
Jul 30 11:14:05.258288 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.258272    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.261194 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.261171    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-228d6060-kserve-self-signed-certs\""
Jul 30 11:14:05.264752 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.264724    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:14:05.277142 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.277116    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"]
Jul 30 11:14:05.279846 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.279813    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-model-cache\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.279972 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.279883    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-dshm\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.280035 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.279981    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-home\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.280125 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.280037    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4677026-8c3a-4bbe-9758-c362e87128ed-tls-certs\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.280125 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.280070    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpbk\" (UniqueName: \"kubernetes.io/projected/d4677026-8c3a-4bbe-9758-c362e87128ed-kube-api-access-kvpbk\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.280236 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.280148    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-tmp-dir\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.380713 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.380677    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-tmp-dir\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.380865 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.380723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-model-cache\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.380865 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.380761    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-dshm\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.380865 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.380790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-home\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.380865 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.380848    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4677026-8c3a-4bbe-9758-c362e87128ed-tls-certs\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.381041 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.380901    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpbk\" (UniqueName: \"kubernetes.io/projected/d4677026-8c3a-4bbe-9758-c362e87128ed-kube-api-access-kvpbk\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.381164 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.381139    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-tmp-dir\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.381261 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.381187    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-model-cache\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.381261 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.381210    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-home\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.383080 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.383057    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-dshm\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.383313 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.383295    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4677026-8c3a-4bbe-9758-c362e87128ed-tls-certs\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.388968 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.388941    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpbk\" (UniqueName: \"kubernetes.io/projected/d4677026-8c3a-4bbe-9758-c362e87128ed-kube-api-access-kvpbk\") pod \"e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") " pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.567444 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.567357    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:05.726231 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:05.726205    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"]
Jul 30 11:14:05.728498 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:14:05.728464    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4677026_8c3a_4bbe_9758_c362e87128ed.slice/crio-a3fd0c740b04438122fadea064bc0eb100bdfee9d133e3c5f1947b39e3aa50cd WatchSource:0}: Error finding container a3fd0c740b04438122fadea064bc0eb100bdfee9d133e3c5f1947b39e3aa50cd: Status 404 returned error can't find the container with id a3fd0c740b04438122fadea064bc0eb100bdfee9d133e3c5f1947b39e3aa50cd
Jul 30 11:14:06.588074 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:06.588024    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq" event={"ID":"d4677026-8c3a-4bbe-9758-c362e87128ed","Type":"ContainerStarted","Data":"c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203"}
Jul 30 11:14:06.588074 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:06.588081    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq" event={"ID":"d4677026-8c3a-4bbe-9758-c362e87128ed","Type":"ContainerStarted","Data":"a3fd0c740b04438122fadea064bc0eb100bdfee9d133e3c5f1947b39e3aa50cd"}
Jul 30 11:14:06.609855 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:06.609803    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq" podStartSLOduration=1.609789731 podStartE2EDuration="1.609789731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:14:06.6084676 +0000 UTC m=+1552.329351862" watchObservedRunningTime="2026-07-30 11:14:06.609789731 +0000 UTC m=+1552.330673992"
Jul 30 11:14:08.193467 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:08.193438    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:14:08.496490 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:08.496404    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:14:10.360503 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.360456    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:14:10.455477 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.455440    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:14:10.676587 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.676512    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"]
Jul 30 11:14:10.680086 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.680063    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.683965 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.683939    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-228d6060-kserve-self-signed-certs\""
Jul 30 11:14:10.693509 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.693482    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"]
Jul 30 11:14:10.702593 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.702565    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:14:10.726399 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.726362    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-model-cache\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.726561 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.726410    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-dshm\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.726561 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.726426    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-tmp-dir\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.726561 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.726505    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79ff7f1-bc1d-4009-a088-3e979022fce9-tls-certs\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.726561 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.726541    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-home\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.726706 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.726573    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxs45\" (UniqueName: \"kubernetes.io/projected/e79ff7f1-bc1d-4009-a088-3e979022fce9-kube-api-access-xxs45\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.827113 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.827077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-dshm\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.827275 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.827131    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-tmp-dir\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.827275 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.827176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79ff7f1-bc1d-4009-a088-3e979022fce9-tls-certs\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.827275 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.827200    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-home\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.827275 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.827230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxs45\" (UniqueName: \"kubernetes.io/projected/e79ff7f1-bc1d-4009-a088-3e979022fce9-kube-api-access-xxs45\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.827275 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.827251    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-model-cache\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.827604 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.827574    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-tmp-dir\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.827604 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.827589    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-home\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.827765 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.827656    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-model-cache\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.829369 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.829351    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-dshm\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.829707 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.829687    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79ff7f1-bc1d-4009-a088-3e979022fce9-tls-certs\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.837551 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.837528    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxs45\" (UniqueName: \"kubernetes.io/projected/e79ff7f1-bc1d-4009-a088-3e979022fce9-kube-api-access-xxs45\") pod \"e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") " pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:10.991324 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:10.991293    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:11.120476 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:11.120444    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"]
Jul 30 11:14:11.123292 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:14:11.123261    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode79ff7f1_bc1d_4009_a088_3e979022fce9.slice/crio-63562f4aa640aa0bb91cebe413a2ecdbad91f22c91996d69bda1e32010d7e1cc WatchSource:0}: Error finding container 63562f4aa640aa0bb91cebe413a2ecdbad91f22c91996d69bda1e32010d7e1cc: Status 404 returned error can't find the container with id 63562f4aa640aa0bb91cebe413a2ecdbad91f22c91996d69bda1e32010d7e1cc
Jul 30 11:14:11.614340 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:11.614299    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp" event={"ID":"e79ff7f1-bc1d-4009-a088-3e979022fce9","Type":"ContainerStarted","Data":"6b0eee3550691d8944fa991a6c592304a4661771275fa39be918c87d0ac148c2"}
Jul 30 11:14:11.614752 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:11.614346    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp" event={"ID":"e79ff7f1-bc1d-4009-a088-3e979022fce9","Type":"ContainerStarted","Data":"63562f4aa640aa0bb91cebe413a2ecdbad91f22c91996d69bda1e32010d7e1cc"}
Jul 30 11:14:11.637880 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:11.637831    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp" podStartSLOduration=1.637816132 podStartE2EDuration="1.637816132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:14:11.637165378 +0000 UTC m=+1557.358049640" watchObservedRunningTime="2026-07-30 11:14:11.637816132 +0000 UTC m=+1557.358700422"
Jul 30 11:14:15.567640 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:15.567545    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:15.567640 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:15.567614    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:15.579870 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:15.579838    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:15.641201 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:15.641176    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:14:18.384516 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:18.384485    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:14:20.991550 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:20.991513    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:20.991922 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:20.991564    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:21.003948 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:21.003927    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:14:21.664294 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:14:21.664264    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:15:06.305197 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:06.305087    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"]
Jul 30 11:15:06.322062 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:06.322036    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:15:07.641375 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:07.641341    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"]
Jul 30 11:15:07.659117 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:07.659072    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:15:08.022442 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:08.022403    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:15:19.115013 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:19.114978    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:15:21.323643 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.323603    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq" podUID="d4677026-8c3a-4bbe-9758-c362e87128ed" containerName="main" containerID="cri-o://c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203" gracePeriod=45
Jul 30 11:15:21.569881 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.569858    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:15:21.702392 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.702362    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-dshm\") pod \"d4677026-8c3a-4bbe-9758-c362e87128ed\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") "
Jul 30 11:15:21.702541 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.702404    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvpbk\" (UniqueName: \"kubernetes.io/projected/d4677026-8c3a-4bbe-9758-c362e87128ed-kube-api-access-kvpbk\") pod \"d4677026-8c3a-4bbe-9758-c362e87128ed\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") "
Jul 30 11:15:21.702541 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.702444    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-model-cache\") pod \"d4677026-8c3a-4bbe-9758-c362e87128ed\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") "
Jul 30 11:15:21.702541 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.702459    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-tmp-dir\") pod \"d4677026-8c3a-4bbe-9758-c362e87128ed\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") "
Jul 30 11:15:21.702541 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.702485    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4677026-8c3a-4bbe-9758-c362e87128ed-tls-certs\") pod \"d4677026-8c3a-4bbe-9758-c362e87128ed\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") "
Jul 30 11:15:21.702747 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.702547    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-home\") pod \"d4677026-8c3a-4bbe-9758-c362e87128ed\" (UID: \"d4677026-8c3a-4bbe-9758-c362e87128ed\") "
Jul 30 11:15:21.702747 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.702737    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4677026-8c3a-4bbe-9758-c362e87128ed" (UID: "d4677026-8c3a-4bbe-9758-c362e87128ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:15:21.702855 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.702761    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4677026-8c3a-4bbe-9758-c362e87128ed" (UID: "d4677026-8c3a-4bbe-9758-c362e87128ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:15:21.702966 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.702925    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-home" (OuterVolumeSpecName: "home") pod "d4677026-8c3a-4bbe-9758-c362e87128ed" (UID: "d4677026-8c3a-4bbe-9758-c362e87128ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:15:21.704604 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.704573    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-dshm" (OuterVolumeSpecName: "dshm") pod "d4677026-8c3a-4bbe-9758-c362e87128ed" (UID: "d4677026-8c3a-4bbe-9758-c362e87128ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:15:21.704692 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.704608    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4677026-8c3a-4bbe-9758-c362e87128ed-kube-api-access-kvpbk" (OuterVolumeSpecName: "kube-api-access-kvpbk") pod "d4677026-8c3a-4bbe-9758-c362e87128ed" (UID: "d4677026-8c3a-4bbe-9758-c362e87128ed"). InnerVolumeSpecName "kube-api-access-kvpbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:15:21.704692 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.704644    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4677026-8c3a-4bbe-9758-c362e87128ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4677026-8c3a-4bbe-9758-c362e87128ed" (UID: "d4677026-8c3a-4bbe-9758-c362e87128ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:15:21.803986 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.803959    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:21.803986 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.803981    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:21.803986 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.803991    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4677026-8c3a-4bbe-9758-c362e87128ed-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:21.804214 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.803999    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:21.804214 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.804006    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4677026-8c3a-4bbe-9758-c362e87128ed-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:21.804214 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.804013    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvpbk\" (UniqueName: \"kubernetes.io/projected/d4677026-8c3a-4bbe-9758-c362e87128ed-kube-api-access-kvpbk\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:21.887966 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.887938    2571 generic.go:356] "Generic (PLEG): container finished" podID="d4677026-8c3a-4bbe-9758-c362e87128ed" containerID="c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203" exitCode=0
Jul 30 11:15:21.888171 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.887980    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"
Jul 30 11:15:21.888171 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.888027    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq" event={"ID":"d4677026-8c3a-4bbe-9758-c362e87128ed","Type":"ContainerDied","Data":"c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203"}
Jul 30 11:15:21.888171 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.888065    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq" event={"ID":"d4677026-8c3a-4bbe-9758-c362e87128ed","Type":"ContainerDied","Data":"a3fd0c740b04438122fadea064bc0eb100bdfee9d133e3c5f1947b39e3aa50cd"}
Jul 30 11:15:21.888171 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.888082    2571 scope.go:117] "RemoveContainer" containerID="c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203"
Jul 30 11:15:21.896904 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.896883    2571 scope.go:117] "RemoveContainer" containerID="c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203"
Jul 30 11:15:21.897178 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:15:21.897161    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203\": container with ID starting with c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203 not found: ID does not exist" containerID="c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203"
Jul 30 11:15:21.897250 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.897192    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203"} err="failed to get container status \"c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203\": rpc error: code = NotFound desc = could not find container \"c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203\": container with ID starting with c8b722e0f86a4bbe85234814fcaa304b127c1860f86e201420235da8fcdf0203 not found: ID does not exist"
Jul 30 11:15:21.920312 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.920289    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"]
Jul 30 11:15:21.924844 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:21.924824    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-228d6060-kserve-6c67949655-c7wmq"]
Jul 30 11:15:22.659440 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:22.659392    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp" podUID="e79ff7f1-bc1d-4009-a088-3e979022fce9" containerName="main" containerID="cri-o://6b0eee3550691d8944fa991a6c592304a4661771275fa39be918c87d0ac148c2" gracePeriod=45
Jul 30 11:15:22.854527 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:22.854496    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4677026-8c3a-4bbe-9758-c362e87128ed" path="/var/lib/kubelet/pods/d4677026-8c3a-4bbe-9758-c362e87128ed/volumes"
Jul 30 11:15:22.894808 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:22.894784    2571 generic.go:356] "Generic (PLEG): container finished" podID="e79ff7f1-bc1d-4009-a088-3e979022fce9" containerID="6b0eee3550691d8944fa991a6c592304a4661771275fa39be918c87d0ac148c2" exitCode=0
Jul 30 11:15:22.894907 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:22.894840    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp" event={"ID":"e79ff7f1-bc1d-4009-a088-3e979022fce9","Type":"ContainerDied","Data":"6b0eee3550691d8944fa991a6c592304a4661771275fa39be918c87d0ac148c2"}
Jul 30 11:15:22.894907 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:22.894863    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp" event={"ID":"e79ff7f1-bc1d-4009-a088-3e979022fce9","Type":"ContainerDied","Data":"63562f4aa640aa0bb91cebe413a2ecdbad91f22c91996d69bda1e32010d7e1cc"}
Jul 30 11:15:22.894907 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:22.894876    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63562f4aa640aa0bb91cebe413a2ecdbad91f22c91996d69bda1e32010d7e1cc"
Jul 30 11:15:22.905582 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:22.905561    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:15:23.014347 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014319    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-model-cache\") pod \"e79ff7f1-bc1d-4009-a088-3e979022fce9\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") "
Jul 30 11:15:23.014578 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014368    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-home\") pod \"e79ff7f1-bc1d-4009-a088-3e979022fce9\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") "
Jul 30 11:15:23.014578 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014401    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxs45\" (UniqueName: \"kubernetes.io/projected/e79ff7f1-bc1d-4009-a088-3e979022fce9-kube-api-access-xxs45\") pod \"e79ff7f1-bc1d-4009-a088-3e979022fce9\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") "
Jul 30 11:15:23.014578 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014446    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-dshm\") pod \"e79ff7f1-bc1d-4009-a088-3e979022fce9\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") "
Jul 30 11:15:23.014578 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014489    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-tmp-dir\") pod \"e79ff7f1-bc1d-4009-a088-3e979022fce9\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") "
Jul 30 11:15:23.014578 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014536    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79ff7f1-bc1d-4009-a088-3e979022fce9-tls-certs\") pod \"e79ff7f1-bc1d-4009-a088-3e979022fce9\" (UID: \"e79ff7f1-bc1d-4009-a088-3e979022fce9\") "
Jul 30 11:15:23.014842 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014643    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e79ff7f1-bc1d-4009-a088-3e979022fce9" (UID: "e79ff7f1-bc1d-4009-a088-3e979022fce9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:15:23.014842 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014686    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-home" (OuterVolumeSpecName: "home") pod "e79ff7f1-bc1d-4009-a088-3e979022fce9" (UID: "e79ff7f1-bc1d-4009-a088-3e979022fce9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:15:23.014842 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014800    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:23.014842 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014818    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:23.014842 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.014817    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e79ff7f1-bc1d-4009-a088-3e979022fce9" (UID: "e79ff7f1-bc1d-4009-a088-3e979022fce9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:15:23.016671 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.016640    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79ff7f1-bc1d-4009-a088-3e979022fce9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e79ff7f1-bc1d-4009-a088-3e979022fce9" (UID: "e79ff7f1-bc1d-4009-a088-3e979022fce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:15:23.016671 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.016640    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-dshm" (OuterVolumeSpecName: "dshm") pod "e79ff7f1-bc1d-4009-a088-3e979022fce9" (UID: "e79ff7f1-bc1d-4009-a088-3e979022fce9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:15:23.016810 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.016713    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79ff7f1-bc1d-4009-a088-3e979022fce9-kube-api-access-xxs45" (OuterVolumeSpecName: "kube-api-access-xxs45") pod "e79ff7f1-bc1d-4009-a088-3e979022fce9" (UID: "e79ff7f1-bc1d-4009-a088-3e979022fce9"). InnerVolumeSpecName "kube-api-access-xxs45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:15:23.115361 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.115319    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:23.115361 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.115356    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79ff7f1-bc1d-4009-a088-3e979022fce9-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:23.115361 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.115367    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79ff7f1-bc1d-4009-a088-3e979022fce9-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:23.115361 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.115376    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxs45\" (UniqueName: \"kubernetes.io/projected/e79ff7f1-bc1d-4009-a088-3e979022fce9-kube-api-access-xxs45\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:15:23.899689 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.899653    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"
Jul 30 11:15:23.924260 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.924230    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"]
Jul 30 11:15:23.934768 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:23.934743    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-228d6060-kserve-6ff5c9b8cf-jtccp"]
Jul 30 11:15:24.853875 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:24.853833    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e79ff7f1-bc1d-4009-a088-3e979022fce9" path="/var/lib/kubelet/pods/e79ff7f1-bc1d-4009-a088-3e979022fce9/volumes"
Jul 30 11:15:25.812919 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:25.812883    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:15:34.758807 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:34.758768    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:15:40.110723 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:40.110688    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:15:47.615436 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:47.615397    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:15:57.395763 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:15:57.395730    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:16:04.099187 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:16:04.099146    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:16:17.235444 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:16:17.235409    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:16:23.284631 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:16:23.284587    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:16:31.182737 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:16:31.182703    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:16:39.286611 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:16:39.286527    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:16:44.490338 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:16:44.490305    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:16:57.797369 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:16:57.797334    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:17:02.892498 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:17:02.892460    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:17:10.090018 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:17:10.089981    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:17:14.978535 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:17:14.978498    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:17:21.386479 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:17:21.386444    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:17:30.684709 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:17:30.684666    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:17:37.699742 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:17:37.699708    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:17:47.697224 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:17:47.697191    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:17:54.789152 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:17:54.789059    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:18:04.083339 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:04.083304    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:18:10.392046 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:10.391948    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:18:19.587145 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:19.587088    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:18:25.788969 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:25.788928    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:18:29.477400 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:29.477356    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:18:31.998460 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:31.998421    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:18:44.497147 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:44.497092    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:18:52.790977 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:52.790942    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:18:56.046411 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:56.046373    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:18:58.489593 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:58.489558    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:18:59.184770 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:59.184734    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:18:59.288240 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:59.288196    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:18:59.524561 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:18:59.524466    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:05.423479 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:05.423445    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:05.491633 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:05.491603    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:05.654516 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:05.654485    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:05.951335 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:05.951305    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:06.350614 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:06.350538    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:07.922579 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:07.922524    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:08.023597 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:08.023564    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:09.054869 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:09.054838    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:09.319935 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:09.319831    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:09.557013 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:09.556980    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:09.636835 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:09.636753    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:12.791167 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:12.791134    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:19:23.338140 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:23.338083    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:33.479425 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.479389    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"]
Jul 30 11:19:33.479837 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.479664    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79ff7f1-bc1d-4009-a088-3e979022fce9" containerName="main"
Jul 30 11:19:33.479837 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.479675    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79ff7f1-bc1d-4009-a088-3e979022fce9" containerName="main"
Jul 30 11:19:33.479837 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.479691    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4677026-8c3a-4bbe-9758-c362e87128ed" containerName="main"
Jul 30 11:19:33.479837 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.479696    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4677026-8c3a-4bbe-9758-c362e87128ed" containerName="main"
Jul 30 11:19:33.479837 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.479741    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79ff7f1-bc1d-4009-a088-3e979022fce9" containerName="main"
Jul 30 11:19:33.479837 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.479748    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4677026-8c3a-4bbe-9758-c362e87128ed" containerName="main"
Jul 30 11:19:33.482484 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.482469    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"
Jul 30 11:19:33.485259 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.485235    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\""
Jul 30 11:19:33.485384 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.485270    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 11:19:33.490893 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.490866    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"]
Jul 30 11:19:33.651668 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.651631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7130e02e-faa4-493b-ac90-c1a012a5e036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc\" (UID: \"7130e02e-faa4-493b-ac90-c1a012a5e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"
Jul 30 11:19:33.752551 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.752471    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7130e02e-faa4-493b-ac90-c1a012a5e036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc\" (UID: \"7130e02e-faa4-493b-ac90-c1a012a5e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"
Jul 30 11:19:33.753085 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.753063    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7130e02e-faa4-493b-ac90-c1a012a5e036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc\" (UID: \"7130e02e-faa4-493b-ac90-c1a012a5e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"
Jul 30 11:19:33.793209 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.793188    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"
Jul 30 11:19:33.922818 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.922794    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"]
Jul 30 11:19:33.924879 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:19:33.924837    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7130e02e_faa4_493b_ac90_c1a012a5e036.slice/crio-e4bb14ef0b09226a2f42f4ffbe675c538e2f4a8f50f9f4e88d783a1a77af43e4 WatchSource:0}: Error finding container e4bb14ef0b09226a2f42f4ffbe675c538e2f4a8f50f9f4e88d783a1a77af43e4: Status 404 returned error can't find the container with id e4bb14ef0b09226a2f42f4ffbe675c538e2f4a8f50f9f4e88d783a1a77af43e4
Jul 30 11:19:33.926842 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:33.926827    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 11:19:34.885171 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:34.885130    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc" event={"ID":"7130e02e-faa4-493b-ac90-c1a012a5e036","Type":"ContainerStarted","Data":"e4bb14ef0b09226a2f42f4ffbe675c538e2f4a8f50f9f4e88d783a1a77af43e4"}
Jul 30 11:19:36.894435 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:36.894401    2571 generic.go:356] "Generic (PLEG): container finished" podID="7130e02e-faa4-493b-ac90-c1a012a5e036" containerID="bce4430a2bdcdcecc9f0314d34b1071906525d52c0b8ab6f94d3c2c3a2bb11d3" exitCode=0
Jul 30 11:19:36.894767 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:36.894453    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc" event={"ID":"7130e02e-faa4-493b-ac90-c1a012a5e036","Type":"ContainerDied","Data":"bce4430a2bdcdcecc9f0314d34b1071906525d52c0b8ab6f94d3c2c3a2bb11d3"}
Jul 30 11:19:38.021176 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:38.021154    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"
Jul 30 11:19:38.185772 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:38.185748    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7130e02e-faa4-493b-ac90-c1a012a5e036-maas-api-service-ca\") pod \"7130e02e-faa4-493b-ac90-c1a012a5e036\" (UID: \"7130e02e-faa4-493b-ac90-c1a012a5e036\") "
Jul 30 11:19:38.186115 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:38.186078    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7130e02e-faa4-493b-ac90-c1a012a5e036-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7130e02e-faa4-493b-ac90-c1a012a5e036" (UID: "7130e02e-faa4-493b-ac90-c1a012a5e036"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:19:38.287287 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:38.287238    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7130e02e-faa4-493b-ac90-c1a012a5e036-maas-api-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:19:38.903847 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:38.903814    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc" event={"ID":"7130e02e-faa4-493b-ac90-c1a012a5e036","Type":"ContainerDied","Data":"e4bb14ef0b09226a2f42f4ffbe675c538e2f4a8f50f9f4e88d783a1a77af43e4"}
Jul 30 11:19:38.903997 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:38.903886    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4bb14ef0b09226a2f42f4ffbe675c538e2f4a8f50f9f4e88d783a1a77af43e4"
Jul 30 11:19:38.903997 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:38.903828    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"
Jul 30 11:19:39.524469 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:39.524437    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:39.592998 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:39.592960    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:39.659710 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:39.659678    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:39.694225 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:39.694194    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"]
Jul 30 11:19:39.695623 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:39.695600    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ca37a8f-pkfhc"]
Jul 30 11:19:39.728328 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:39.728306    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:39.842000 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:39.841931    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:40.113603 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:40.113524    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:40.320925 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:40.320898    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:40.392667 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:40.392589    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:40.538723 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:40.538694    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:40.854916 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:40.854884    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7130e02e-faa4-493b-ac90-c1a012a5e036" path="/var/lib/kubelet/pods/7130e02e-faa4-493b-ac90-c1a012a5e036/volumes"
Jul 30 11:19:40.869408 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:40.869379    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:41.082755 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:41.082722    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:42.590214 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:42.590177    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:19:48.855370 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:48.855334    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:49.014504 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:49.014473    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:49.168016 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:49.167931    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:49.513606 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:49.513575    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:49.600744 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:49.600711    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:19:52.293250 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:19:52.293217    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:20:13.843737 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:13.843701    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"]
Jul 30 11:20:13.844201 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:13.843961    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7130e02e-faa4-493b-ac90-c1a012a5e036" containerName="revoke-keys"
Jul 30 11:20:13.844201 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:13.843972    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7130e02e-faa4-493b-ac90-c1a012a5e036" containerName="revoke-keys"
Jul 30 11:20:13.844201 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:13.844028    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7130e02e-faa4-493b-ac90-c1a012a5e036" containerName="revoke-keys"
Jul 30 11:20:13.848078 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:13.848055    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"
Jul 30 11:20:13.850716 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:13.850695    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 11:20:13.850840 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:13.850752    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\""
Jul 30 11:20:13.853840 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:13.853820    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"]
Jul 30 11:20:13.924263 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:13.924229    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62493b5-8dab-49e2-b989-7cfcb0d6728a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4\" (UID: \"c62493b5-8dab-49e2-b989-7cfcb0d6728a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"
Jul 30 11:20:14.025170 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:14.025136    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62493b5-8dab-49e2-b989-7cfcb0d6728a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4\" (UID: \"c62493b5-8dab-49e2-b989-7cfcb0d6728a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"
Jul 30 11:20:14.025773 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:14.025754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62493b5-8dab-49e2-b989-7cfcb0d6728a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4\" (UID: \"c62493b5-8dab-49e2-b989-7cfcb0d6728a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"
Jul 30 11:20:14.159580 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:14.159506    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"
Jul 30 11:20:14.279989 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:14.279959    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"]
Jul 30 11:20:14.282342 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:20:14.282307    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62493b5_8dab_49e2_b989_7cfcb0d6728a.slice/crio-a2e45d55442f94dac45729412d196fe02ec6b4bb3a4cfc48d489be4fd11116f3 WatchSource:0}: Error finding container a2e45d55442f94dac45729412d196fe02ec6b4bb3a4cfc48d489be4fd11116f3: Status 404 returned error can't find the container with id a2e45d55442f94dac45729412d196fe02ec6b4bb3a4cfc48d489be4fd11116f3
Jul 30 11:20:14.840742 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:14.840669    2571 scope.go:117] "RemoveContainer" containerID="6b0eee3550691d8944fa991a6c592304a4661771275fa39be918c87d0ac148c2"
Jul 30 11:20:15.040605 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:15.040567    2571 generic.go:356] "Generic (PLEG): container finished" podID="c62493b5-8dab-49e2-b989-7cfcb0d6728a" containerID="1393695c06423b997e0576f2415d2e696994760a450948987a79146bd007bfa6" exitCode=0
Jul 30 11:20:15.040976 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:15.040656    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4" event={"ID":"c62493b5-8dab-49e2-b989-7cfcb0d6728a","Type":"ContainerDied","Data":"1393695c06423b997e0576f2415d2e696994760a450948987a79146bd007bfa6"}
Jul 30 11:20:15.040976 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:15.040714    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4" event={"ID":"c62493b5-8dab-49e2-b989-7cfcb0d6728a","Type":"ContainerStarted","Data":"a2e45d55442f94dac45729412d196fe02ec6b4bb3a4cfc48d489be4fd11116f3"}
Jul 30 11:20:16.164009 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:16.163947    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"
Jul 30 11:20:16.240984 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:16.240947    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62493b5-8dab-49e2-b989-7cfcb0d6728a-maas-api-service-ca\") pod \"c62493b5-8dab-49e2-b989-7cfcb0d6728a\" (UID: \"c62493b5-8dab-49e2-b989-7cfcb0d6728a\") "
Jul 30 11:20:16.241350 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:16.241325    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62493b5-8dab-49e2-b989-7cfcb0d6728a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c62493b5-8dab-49e2-b989-7cfcb0d6728a" (UID: "c62493b5-8dab-49e2-b989-7cfcb0d6728a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:20:16.342120 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:16.342081    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62493b5-8dab-49e2-b989-7cfcb0d6728a-maas-api-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:20:17.050212 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:17.050163    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4" event={"ID":"c62493b5-8dab-49e2-b989-7cfcb0d6728a","Type":"ContainerDied","Data":"a2e45d55442f94dac45729412d196fe02ec6b4bb3a4cfc48d489be4fd11116f3"}
Jul 30 11:20:17.050469 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:17.050452    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2e45d55442f94dac45729412d196fe02ec6b4bb3a4cfc48d489be4fd11116f3"
Jul 30 11:20:17.050664 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:17.050652    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"
Jul 30 11:20:20.538225 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:20.538185    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:20.636137 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:20.636086    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:20.698419 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:20.698390    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"]
Jul 30 11:20:20.700736 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:20.700714    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9108b46-pf5c4"]
Jul 30 11:20:20.702032 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:20.702013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:20.764166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:20.764139    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:20.854086 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:20.854014    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c62493b5-8dab-49e2-b989-7cfcb0d6728a" path="/var/lib/kubelet/pods/c62493b5-8dab-49e2-b989-7cfcb0d6728a/volumes"
Jul 30 11:20:20.890499 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:20.890468    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:21.092499 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:21.092469    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:21.330864 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:21.330831    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:21.655537 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:21.655462    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:21.772743 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:21.772714    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:22.586998 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:22.586955    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:20:24.316220 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:24.316188    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:24.414339 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:24.414308    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:25.449781 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:25.449749    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:25.549452 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:25.549420    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:26.043939 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:26.043903    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:26.143886 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:26.143855    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:28.884463 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:28.884425    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:20:50.115592 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.115558    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"]
Jul 30 11:20:50.116059 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.115841    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c62493b5-8dab-49e2-b989-7cfcb0d6728a" containerName="revoke-keys"
Jul 30 11:20:50.116059 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.115861    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c62493b5-8dab-49e2-b989-7cfcb0d6728a" containerName="revoke-keys"
Jul 30 11:20:50.116059 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.115910    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c62493b5-8dab-49e2-b989-7cfcb0d6728a" containerName="revoke-keys"
Jul 30 11:20:50.117745 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.117726    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"
Jul 30 11:20:50.120566 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.120545    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\""
Jul 30 11:20:50.120701 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.120553    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 11:20:50.125958 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.125934    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"]
Jul 30 11:20:50.191715 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.191684    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7b543d-e944-45e6-bdee-af673300335e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg\" (UID: \"ad7b543d-e944-45e6-bdee-af673300335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"
Jul 30 11:20:50.292731 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.292689    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7b543d-e944-45e6-bdee-af673300335e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg\" (UID: \"ad7b543d-e944-45e6-bdee-af673300335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"
Jul 30 11:20:50.293443 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.293422    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7b543d-e944-45e6-bdee-af673300335e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg\" (UID: \"ad7b543d-e944-45e6-bdee-af673300335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"
Jul 30 11:20:50.428982 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.428953    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"
Jul 30 11:20:50.553074 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:50.553041    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"]
Jul 30 11:20:50.556435 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:20:50.556404    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7b543d_e944_45e6_bdee_af673300335e.slice/crio-577f45d9b91a6f403355c93cf3e4cb147017a64c34a74f6b29bee9bb90260c69 WatchSource:0}: Error finding container 577f45d9b91a6f403355c93cf3e4cb147017a64c34a74f6b29bee9bb90260c69: Status 404 returned error can't find the container with id 577f45d9b91a6f403355c93cf3e4cb147017a64c34a74f6b29bee9bb90260c69
Jul 30 11:20:51.182138 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:51.182081    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg" event={"ID":"ad7b543d-e944-45e6-bdee-af673300335e","Type":"ContainerStarted","Data":"aa5806ac54eec31643a58294402d722eab401b009c8f54ceb367b975209cd831"}
Jul 30 11:20:51.182512 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:51.182156    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg" event={"ID":"ad7b543d-e944-45e6-bdee-af673300335e","Type":"ContainerStarted","Data":"577f45d9b91a6f403355c93cf3e4cb147017a64c34a74f6b29bee9bb90260c69"}
Jul 30 11:20:51.198253 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:51.198208    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg" podStartSLOduration=1.198194416 podStartE2EDuration="1.198194416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:20:51.196737069 +0000 UTC m=+1956.917621329" watchObservedRunningTime="2026-07-30 11:20:51.198194416 +0000 UTC m=+1956.919078677"
Jul 30 11:20:52.187543 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:52.187508    2571 generic.go:356] "Generic (PLEG): container finished" podID="ad7b543d-e944-45e6-bdee-af673300335e" containerID="aa5806ac54eec31643a58294402d722eab401b009c8f54ceb367b975209cd831" exitCode=0
Jul 30 11:20:52.187925 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:52.187586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg" event={"ID":"ad7b543d-e944-45e6-bdee-af673300335e","Type":"ContainerDied","Data":"aa5806ac54eec31643a58294402d722eab401b009c8f54ceb367b975209cd831"}
Jul 30 11:20:53.311694 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:53.311667    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"
Jul 30 11:20:53.422176 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:53.422146    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7b543d-e944-45e6-bdee-af673300335e-maas-api-service-ca\") pod \"ad7b543d-e944-45e6-bdee-af673300335e\" (UID: \"ad7b543d-e944-45e6-bdee-af673300335e\") "
Jul 30 11:20:53.422500 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:53.422472    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7b543d-e944-45e6-bdee-af673300335e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad7b543d-e944-45e6-bdee-af673300335e" (UID: "ad7b543d-e944-45e6-bdee-af673300335e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:20:53.523564 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:53.523477    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7b543d-e944-45e6-bdee-af673300335e-maas-api-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:20:54.196317 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:54.196283    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg" event={"ID":"ad7b543d-e944-45e6-bdee-af673300335e","Type":"ContainerDied","Data":"577f45d9b91a6f403355c93cf3e4cb147017a64c34a74f6b29bee9bb90260c69"}
Jul 30 11:20:54.196317 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:54.196317    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"
Jul 30 11:20:54.196524 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:54.196340    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="577f45d9b91a6f403355c93cf3e4cb147017a64c34a74f6b29bee9bb90260c69"
Jul 30 11:20:55.956514 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:55.956479    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:56.056030 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:56.055994    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:56.116844 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:56.116815    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"]
Jul 30 11:20:56.119296 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:56.119273    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-519ed3f7-wjxzg"]
Jul 30 11:20:56.159524 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:56.159498    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:56.855175 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:56.855136    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad7b543d-e944-45e6-bdee-af673300335e" path="/var/lib/kubelet/pods/ad7b543d-e944-45e6-bdee-af673300335e/volumes"
Jul 30 11:20:57.266914 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:57.266881    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:57.494073 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:57.494039    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:57.750829 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:57.750794    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:58.045173 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:58.045083    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:58.371117 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:58.371020    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:20:59.190059 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:20:59.190026    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:21:00.205320 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:00.205290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:00.300490 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:00.300447    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:01.424703 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:01.424671    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:01.546061 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:01.546025    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:01.936249 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:01.936221    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:02.017853 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:02.017828    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:08.704983 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:08.704902    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:21:32.057597 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:32.057559    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:32.155811 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:32.155773    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:32.269549 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:32.269509    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:32.358430 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:32.358347    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:32.596249 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:32.596216    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:33.050965 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:33.050925    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:33.146951 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:33.146913    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:33.247167 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:33.247129    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:34.797691 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:34.797651    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:21:44.900898 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:44.900860    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:47.694182 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:47.694145    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:21:51.968383 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:51.968321    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:52.032225 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:52.032191    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:52.128989 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:52.128955    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:52.234123 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:52.234004    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:52.326235 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:52.326201    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:21:52.673080 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:21:52.672992    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:00.490668 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:00.490634    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:18.673614 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:18.673579    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:25.007237 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:25.007198    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:25.113464 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:25.113426    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:25.220343 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:25.220307    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:41.144064 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:41.143972    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:47.351011 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:47.350970    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:47.413394 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:47.413360    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:47.515635 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:47.515602    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:47.667522 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:47.667442    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:22:58.093546 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:22:58.093512    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:09.162878 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:09.162831    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:15.618673 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:15.618635    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:15.686417 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:15.686384    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:15.765432 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:15.765394    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:15.866801 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:15.866769    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:16.558027 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:16.557992    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:16.655424 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:16.655391    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:16.755779 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:16.755747    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:18.299922 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:18.299884    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:22.595221 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:22.595179    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:23:24.972387 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:24.972355    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:25.388522 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:25.388437    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:25.632526 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:25.632498    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:27.390328 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:27.390287    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:23:32.214329 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:32.214292    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:33.776194 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:33.776158    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:35.949291 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:35.949253    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:36.050551 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:36.050516    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:37.730190 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:37.730147    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:38.220203 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:38.220172    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:38.357275 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:38.357246    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:38.423621 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:38.423589    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:40.097062 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:40.097024    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:23:50.670819 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:50.670787    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:51.350904 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:51.350865    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:51.450891 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:51.450861    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:51.779149 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:51.779122    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:58.299645 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.299608    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:58.399700 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.399668    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:23:58.649686 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.649610    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"]
Jul 30 11:23:58.649885 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.649872    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7b543d-e944-45e6-bdee-af673300335e" containerName="revoke-keys"
Jul 30 11:23:58.649947 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.649886    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7b543d-e944-45e6-bdee-af673300335e" containerName="revoke-keys"
Jul 30 11:23:58.649947 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.649941    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7b543d-e944-45e6-bdee-af673300335e" containerName="revoke-keys"
Jul 30 11:23:58.652681 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.652664    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"
Jul 30 11:23:58.655544 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.655528    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 11:23:58.655807 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.655793    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\""
Jul 30 11:23:58.660176 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.660156    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"]
Jul 30 11:23:58.802717 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.802675    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276cc18c-3dae-433c-a45b-01d07b4e533b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-203e4c55-tsphd\" (UID: \"276cc18c-3dae-433c-a45b-01d07b4e533b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"
Jul 30 11:23:58.903182 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.903086    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276cc18c-3dae-433c-a45b-01d07b4e533b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-203e4c55-tsphd\" (UID: \"276cc18c-3dae-433c-a45b-01d07b4e533b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"
Jul 30 11:23:58.903704 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.903682    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276cc18c-3dae-433c-a45b-01d07b4e533b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-203e4c55-tsphd\" (UID: \"276cc18c-3dae-433c-a45b-01d07b4e533b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"
Jul 30 11:23:58.963482 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:58.963450    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"
Jul 30 11:23:59.096227 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:59.096195    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"]
Jul 30 11:23:59.099065 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:23:59.099035    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276cc18c_3dae_433c_a45b_01d07b4e533b.slice/crio-16ced3019f1037f04c6286b17c3564275b7569a24408d96d1e4340ada1b06853 WatchSource:0}: Error finding container 16ced3019f1037f04c6286b17c3564275b7569a24408d96d1e4340ada1b06853: Status 404 returned error can't find the container with id 16ced3019f1037f04c6286b17c3564275b7569a24408d96d1e4340ada1b06853
Jul 30 11:23:59.945805 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:59.945768    2571 generic.go:356] "Generic (PLEG): container finished" podID="276cc18c-3dae-433c-a45b-01d07b4e533b" containerID="b0abcd9a25d126113db75ff2a0e36828dfad8f72f89ef4d63656c47577a4c7bc" exitCode=0
Jul 30 11:23:59.946193 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:59.945832    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd" event={"ID":"276cc18c-3dae-433c-a45b-01d07b4e533b","Type":"ContainerDied","Data":"b0abcd9a25d126113db75ff2a0e36828dfad8f72f89ef4d63656c47577a4c7bc"}
Jul 30 11:23:59.946193 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:23:59.945914    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd" event={"ID":"276cc18c-3dae-433c-a45b-01d07b4e533b","Type":"ContainerStarted","Data":"16ced3019f1037f04c6286b17c3564275b7569a24408d96d1e4340ada1b06853"}
Jul 30 11:24:01.070252 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:01.070227    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"
Jul 30 11:24:01.222832 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:01.222751    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276cc18c-3dae-433c-a45b-01d07b4e533b-maas-api-service-ca\") pod \"276cc18c-3dae-433c-a45b-01d07b4e533b\" (UID: \"276cc18c-3dae-433c-a45b-01d07b4e533b\") "
Jul 30 11:24:01.223147 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:01.223127    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276cc18c-3dae-433c-a45b-01d07b4e533b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "276cc18c-3dae-433c-a45b-01d07b4e533b" (UID: "276cc18c-3dae-433c-a45b-01d07b4e533b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:24:01.324058 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:01.324026    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276cc18c-3dae-433c-a45b-01d07b4e533b-maas-api-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:24:01.954551 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:01.954526    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"
Jul 30 11:24:01.954718 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:01.954520    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd" event={"ID":"276cc18c-3dae-433c-a45b-01d07b4e533b","Type":"ContainerDied","Data":"16ced3019f1037f04c6286b17c3564275b7569a24408d96d1e4340ada1b06853"}
Jul 30 11:24:01.954718 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:01.954664    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16ced3019f1037f04c6286b17c3564275b7569a24408d96d1e4340ada1b06853"
Jul 30 11:24:02.961361 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:02.961329    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:03.040986 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:03.040960    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:03.125652 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:03.125617    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"]
Jul 30 11:24:03.130076 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:03.130051    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-203e4c55-tsphd"]
Jul 30 11:24:03.164832 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:03.164807    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:03.509112 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:03.509063    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:04.196185 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:04.196153    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:24:04.853977 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:04.853942    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276cc18c-3dae-433c-a45b-01d07b4e533b" path="/var/lib/kubelet/pods/276cc18c-3dae-433c-a45b-01d07b4e533b/volumes"
Jul 30 11:24:05.063756 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:05.063730    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:09.773767 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:09.773694    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:09.830293 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:09.830263    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:09.932190 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:09.932159    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:10.146049 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:10.145975    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:10.259448 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:10.259418    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:10.486389 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:10.486360    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:10.586553 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:10.586519    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:16.376013 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:16.375978    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:16.440363 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:16.440334    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:16.502630 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:16.502599    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:16.608900 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:16.608870    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:16.731249 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:16.731221    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:22.777340 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:22.777304    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:22.839560 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:22.839532    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:24:44.188659 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:24:44.188623    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:25:12.705402 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:25:12.705363    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:25:12.805723 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:25:12.805693    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:08.526769 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:08.526702    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:14.886939 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:14.886911    2571 scope.go:117] "RemoveContainer" containerID="bce4430a2bdcdcecc9f0314d34b1071906525d52c0b8ab6f94d3c2c3a2bb11d3"
Jul 30 11:26:14.894308 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:14.894290    2571 scope.go:117] "RemoveContainer" containerID="1393695c06423b997e0576f2415d2e696994760a450948987a79146bd007bfa6"
Jul 30 11:26:25.300179 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:25.300143    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:25.619415 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:25.619334    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:25.698679 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:25.698648    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:26:25.821692 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:25.821660    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:28.859897 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:28.859860    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:28.927974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:28.927945    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:29.972637 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:29.972587    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:30.263447 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:30.263365    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:30.479055 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:30.479020    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:30.580703 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:30.580625    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:26:33.086663 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:33.086628    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:26:53.839748 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:26:53.839712    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:27:08.920126 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.920032    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"]
Jul 30 11:27:08.920490 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.920291    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276cc18c-3dae-433c-a45b-01d07b4e533b" containerName="revoke-keys"
Jul 30 11:27:08.920490 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.920302    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="276cc18c-3dae-433c-a45b-01d07b4e533b" containerName="revoke-keys"
Jul 30 11:27:08.920490 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.920349    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="276cc18c-3dae-433c-a45b-01d07b4e533b" containerName="revoke-keys"
Jul 30 11:27:08.922943 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.922925    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:08.925975 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.925950    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6bccc947\"/\"default-dockercfg-9qfpk\""
Jul 30 11:27:08.926484 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.926463    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6bccc947\"/\"kube-root-ca.crt\""
Jul 30 11:27:08.927302 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.927285    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6bccc947\"/\"e2e-lifecycle-model-6bccc947-kserve-self-signed-certs\""
Jul 30 11:27:08.927397 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.927327    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6bccc947\"/\"openshift-service-ca.crt\""
Jul 30 11:27:08.936129 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.936090    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"]
Jul 30 11:27:08.994756 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:08.994722    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:27:09.012950 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.012920    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v65jw\" (UniqueName: \"kubernetes.io/projected/247f1160-b885-462e-9b20-428780ca307d-kube-api-access-v65jw\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.013077 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.012954    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-model-cache\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.013077 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.013012    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-tmp-dir\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.013077 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.013048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-dshm\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.013077 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.013065    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247f1160-b885-462e-9b20-428780ca307d-tls-certs\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.013234 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.013114    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-home\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.113457 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.113427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-tmp-dir\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.113597 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.113466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-dshm\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.113597 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.113486    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247f1160-b885-462e-9b20-428780ca307d-tls-certs\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.113597 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.113524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-home\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.113597 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.113559    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v65jw\" (UniqueName: \"kubernetes.io/projected/247f1160-b885-462e-9b20-428780ca307d-kube-api-access-v65jw\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.113597 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.113587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-model-cache\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.113864 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.113841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-tmp-dir\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.113940 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.113909    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-home\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.113990 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.113946    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-model-cache\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.115723 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.115702    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-dshm\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.115980 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.115963    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247f1160-b885-462e-9b20-428780ca307d-tls-certs\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.125484 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.125460    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v65jw\" (UniqueName: \"kubernetes.io/projected/247f1160-b885-462e-9b20-428780ca307d-kube-api-access-v65jw\") pod \"e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") " pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.231545 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.231513    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:09.359874 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.359842    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"]
Jul 30 11:27:09.362113 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:27:09.362079    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod247f1160_b885_462e_9b20_428780ca307d.slice/crio-096d1dc33e1a12b7d6457075392a77c81abf117a8013fd23cbce760c54b8d160 WatchSource:0}: Error finding container 096d1dc33e1a12b7d6457075392a77c81abf117a8013fd23cbce760c54b8d160: Status 404 returned error can't find the container with id 096d1dc33e1a12b7d6457075392a77c81abf117a8013fd23cbce760c54b8d160
Jul 30 11:27:09.704314 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.704280    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn" event={"ID":"247f1160-b885-462e-9b20-428780ca307d","Type":"ContainerStarted","Data":"22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928"}
Jul 30 11:27:09.704314 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.704318    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn" event={"ID":"247f1160-b885-462e-9b20-428780ca307d","Type":"ContainerStarted","Data":"096d1dc33e1a12b7d6457075392a77c81abf117a8013fd23cbce760c54b8d160"}
Jul 30 11:27:09.723367 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:09.723321    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn" podStartSLOduration=1.723305879 podStartE2EDuration="1.723305879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:27:09.722429815 +0000 UTC m=+2335.443314158" watchObservedRunningTime="2026-07-30 11:27:09.723305879 +0000 UTC m=+2335.444190140"
Jul 30 11:27:11.159458 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:11.159423    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:27:14.912571 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:14.912541    2571 scope.go:117] "RemoveContainer" containerID="aa5806ac54eec31643a58294402d722eab401b009c8f54ceb367b975209cd831"
Jul 30 11:27:19.231974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:19.231933    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:19.231974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:19.231982    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:19.244160 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:19.244134    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:19.756073 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:19.756045    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:27:23.097339 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:23.097301    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:27:40.609166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:40.609125    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:27:43.115913 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:43.115832    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:27:43.216583 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:43.216553    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:27:43.309115 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:43.309077    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:27:43.370817 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:43.370744    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:27:46.284381 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:46.284344    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:27:46.499242 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:46.499204    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:27:51.461080 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:51.461048    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"]
Jul 30 11:27:51.559456 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:27:51.559353    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:06.479527 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.479469    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn" podUID="247f1160-b885-462e-9b20-428780ca307d" containerName="main" containerID="cri-o://22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928" gracePeriod=45
Jul 30 11:28:06.717164 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.717138    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:28:06.853075 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.852982    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v65jw\" (UniqueName: \"kubernetes.io/projected/247f1160-b885-462e-9b20-428780ca307d-kube-api-access-v65jw\") pod \"247f1160-b885-462e-9b20-428780ca307d\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") "
Jul 30 11:28:06.853075 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853018    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247f1160-b885-462e-9b20-428780ca307d-tls-certs\") pod \"247f1160-b885-462e-9b20-428780ca307d\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") "
Jul 30 11:28:06.853075 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853053    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-dshm\") pod \"247f1160-b885-462e-9b20-428780ca307d\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") "
Jul 30 11:28:06.853075 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853073    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-home\") pod \"247f1160-b885-462e-9b20-428780ca307d\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") "
Jul 30 11:28:06.853427 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853115    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-tmp-dir\") pod \"247f1160-b885-462e-9b20-428780ca307d\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") "
Jul 30 11:28:06.853427 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853142    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-model-cache\") pod \"247f1160-b885-462e-9b20-428780ca307d\" (UID: \"247f1160-b885-462e-9b20-428780ca307d\") "
Jul 30 11:28:06.853427 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853336    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-home" (OuterVolumeSpecName: "home") pod "247f1160-b885-462e-9b20-428780ca307d" (UID: "247f1160-b885-462e-9b20-428780ca307d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:28:06.853427 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853379    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "247f1160-b885-462e-9b20-428780ca307d" (UID: "247f1160-b885-462e-9b20-428780ca307d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:28:06.853636 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853454    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-model-cache" (OuterVolumeSpecName: "model-cache") pod "247f1160-b885-462e-9b20-428780ca307d" (UID: "247f1160-b885-462e-9b20-428780ca307d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:28:06.853692 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853648    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:28:06.853692 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853668    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:28:06.853692 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.853684    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:28:06.855268 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.855238    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-dshm" (OuterVolumeSpecName: "dshm") pod "247f1160-b885-462e-9b20-428780ca307d" (UID: "247f1160-b885-462e-9b20-428780ca307d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:28:06.855400 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.855360    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247f1160-b885-462e-9b20-428780ca307d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "247f1160-b885-462e-9b20-428780ca307d" (UID: "247f1160-b885-462e-9b20-428780ca307d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:28:06.855540 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.855524    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247f1160-b885-462e-9b20-428780ca307d-kube-api-access-v65jw" (OuterVolumeSpecName: "kube-api-access-v65jw") pod "247f1160-b885-462e-9b20-428780ca307d" (UID: "247f1160-b885-462e-9b20-428780ca307d"). InnerVolumeSpecName "kube-api-access-v65jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:28:06.932247 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.932215    2571 generic.go:356] "Generic (PLEG): container finished" podID="247f1160-b885-462e-9b20-428780ca307d" containerID="22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928" exitCode=0
Jul 30 11:28:06.932391 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.932267    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"
Jul 30 11:28:06.932391 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.932304    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn" event={"ID":"247f1160-b885-462e-9b20-428780ca307d","Type":"ContainerDied","Data":"22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928"}
Jul 30 11:28:06.932391 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.932348    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn" event={"ID":"247f1160-b885-462e-9b20-428780ca307d","Type":"ContainerDied","Data":"096d1dc33e1a12b7d6457075392a77c81abf117a8013fd23cbce760c54b8d160"}
Jul 30 11:28:06.932391 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.932369    2571 scope.go:117] "RemoveContainer" containerID="22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928"
Jul 30 11:28:06.940213 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.940188    2571 scope.go:117] "RemoveContainer" containerID="22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928"
Jul 30 11:28:06.940476 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:28:06.940456    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928\": container with ID starting with 22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928 not found: ID does not exist" containerID="22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928"
Jul 30 11:28:06.940529 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.940483    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928"} err="failed to get container status \"22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928\": rpc error: code = NotFound desc = could not find container \"22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928\": container with ID starting with 22cc1ce131bd2c0179e25dec176d5f3204152cdc15706b31ec31e60c3e8eb928 not found: ID does not exist"
Jul 30 11:28:06.954159 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.954136    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247f1160-b885-462e-9b20-428780ca307d-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:28:06.954159 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.954158    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v65jw\" (UniqueName: \"kubernetes.io/projected/247f1160-b885-462e-9b20-428780ca307d-kube-api-access-v65jw\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:28:06.954320 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.954171    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247f1160-b885-462e-9b20-428780ca307d-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:28:06.968653 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.968632    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"]
Jul 30 11:28:06.972729 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:06.972709    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6bccc947/e2e-lifecycle-model-6bccc947-kserve-5ccdf4b675-hbwdn"]
Jul 30 11:28:08.854363 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:08.854328    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247f1160-b885-462e-9b20-428780ca307d" path="/var/lib/kubelet/pods/247f1160-b885-462e-9b20-428780ca307d/volumes"
Jul 30 11:28:27.919082 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:27.919046    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:29.116021 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:29.115988    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:35.946262 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:35.946224    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:38.354384 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:38.354350    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:28:39.760730 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:39.760693    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:46.601501 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:46.601457    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:47.961168 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:47.961133    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:48.076771 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:48.076742    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:48.918844 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:48.918814    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:49.118759 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:49.118730    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:50.534409 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:50.534369    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:50.705855 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:50.705825    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:50.944216 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:50.944186    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:28:59.123859 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:59.123822    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:59.226776 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:59.226746    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:28:59.540637 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:59.540604    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:28:59.911160 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:28:59.911065    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:06.422112 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:06.422070    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:29:08.055435 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:08.055401    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:08.247691 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:08.247656    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:08.349684 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:08.349603    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:17.841642 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:17.841610    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:18.869406 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:18.869370    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:19.126558 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:19.126481    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:40.062295 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:40.062260    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:46.995166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:46.995126    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:47.133378 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:47.133313    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:47.374887 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:47.374801    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:47.690202 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:47.690173    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:48.016257 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:48.016175    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:50.961524 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:50.961495    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:51.096479 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:51.096451    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:52.086190 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:52.086158    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:52.334680 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:52.334647    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:52.703448 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:52.703411    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:52.803637 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:52.803605    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:29:55.309508 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:29:55.309473    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:30:00.136438 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:00.136405    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756850-29k4f"]
Jul 30 11:30:00.136804 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:00.136687    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247f1160-b885-462e-9b20-428780ca307d" containerName="main"
Jul 30 11:30:00.136804 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:00.136698    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="247f1160-b885-462e-9b20-428780ca307d" containerName="main"
Jul 30 11:30:00.136804 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:00.136752    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="247f1160-b885-462e-9b20-428780ca307d" containerName="main"
Jul 30 11:30:00.139766 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:00.139749    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756850-29k4f"
Jul 30 11:30:00.142748 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:00.142730    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\""
Jul 30 11:30:00.149941 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:00.149905    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756850-29k4f"]
Jul 30 11:30:00.150248 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:00.150224    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756850-29k4f"
Jul 30 11:30:00.280043 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:00.280017    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756850-29k4f"]
Jul 30 11:30:00.282361 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:30:00.282324    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f363fa_8174_4186_99f1_51ef10897688.slice/crio-06cbe28289d0302dcd5057d2266a3f5746c659833d89bfd54e69f7476801bdd8 WatchSource:0}: Error finding container 06cbe28289d0302dcd5057d2266a3f5746c659833d89bfd54e69f7476801bdd8: Status 404 returned error can't find the container with id 06cbe28289d0302dcd5057d2266a3f5746c659833d89bfd54e69f7476801bdd8
Jul 30 11:30:00.384867 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:00.384831    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756850-29k4f" event={"ID":"91f363fa-8174-4186-99f1-51ef10897688","Type":"ContainerStarted","Data":"06cbe28289d0302dcd5057d2266a3f5746c659833d89bfd54e69f7476801bdd8"}
Jul 30 11:30:01.389936 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:01.389840    2571 generic.go:356] "Generic (PLEG): container finished" podID="91f363fa-8174-4186-99f1-51ef10897688" containerID="2844a6af5532f821495a31f3ffb8e1540fb6daa71f1af20dc84bf5f73dccb74f" exitCode=0
Jul 30 11:30:01.389936 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:01.389896    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756850-29k4f" event={"ID":"91f363fa-8174-4186-99f1-51ef10897688","Type":"ContainerDied","Data":"2844a6af5532f821495a31f3ffb8e1540fb6daa71f1af20dc84bf5f73dccb74f"}
Jul 30 11:30:02.517498 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:02.517471    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756850-29k4f"
Jul 30 11:30:03.399940 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:03.399898    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756850-29k4f" event={"ID":"91f363fa-8174-4186-99f1-51ef10897688","Type":"ContainerDied","Data":"06cbe28289d0302dcd5057d2266a3f5746c659833d89bfd54e69f7476801bdd8"}
Jul 30 11:30:03.400123 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:03.399957    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06cbe28289d0302dcd5057d2266a3f5746c659833d89bfd54e69f7476801bdd8"
Jul 30 11:30:03.400123 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:03.399922    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756850-29k4f"
Jul 30 11:30:14.943126 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:14.943040    2571 scope.go:117] "RemoveContainer" containerID="b0abcd9a25d126113db75ff2a0e36828dfad8f72f89ef4d63656c47577a4c7bc"
Jul 30 11:30:15.989243 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:15.989209    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:16.347765 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:16.347681    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:16.664626 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:16.664550    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:17.101513 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:17.101483    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:20.139165 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:20.139132    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:20.239217 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:20.239183    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:21.306067 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:21.306030    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:21.469365 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:21.469328    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:21.831191 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:21.831163    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:21.930604 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:21.930570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:24.501690 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:24.501658    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:30:24.567635 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:24.567603    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:24.645081 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:24.645044    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:32.057733 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:32.057698    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:45.239167 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:45.239136    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:51.880356 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.880324    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"]
Jul 30 11:30:51.880822 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.880583    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f363fa-8174-4186-99f1-51ef10897688" containerName="cleanup"
Jul 30 11:30:51.880822 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.880592    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f363fa-8174-4186-99f1-51ef10897688" containerName="cleanup"
Jul 30 11:30:51.880822 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.880641    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f363fa-8174-4186-99f1-51ef10897688" containerName="cleanup"
Jul 30 11:30:51.883370 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.883354    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:51.887825 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.887794    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-da4b33\"/\"kube-root-ca.crt\""
Jul 30 11:30:51.887825 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.887809    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-da4b33\"/\"openshift-service-ca.crt\""
Jul 30 11:30:51.887825 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.887819    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-da4b33\"/\"default-dockercfg-nbxkf\""
Jul 30 11:30:51.888062 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.887826    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-da4b33\"/\"test-model-da4b33-kserve-self-signed-certs\""
Jul 30 11:30:51.893067 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.893041    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"]
Jul 30 11:30:51.930409 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.930376    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:30:51.988932 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.988906    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-home\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:51.989065 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.988942    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-tmp-dir\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:51.989065 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.988964    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be5f6524-6406-46eb-bf85-8e1ca741a24d-tls-certs\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:51.989065 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.988985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc2fd\" (UniqueName: \"kubernetes.io/projected/be5f6524-6406-46eb-bf85-8e1ca741a24d-kube-api-access-kc2fd\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:51.989065 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.989041    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-dshm\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:51.989240 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:51.989116    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-model-cache\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.090349 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.090318    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-tmp-dir\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.090483 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.090354    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be5f6524-6406-46eb-bf85-8e1ca741a24d-tls-certs\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.090483 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.090374    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc2fd\" (UniqueName: \"kubernetes.io/projected/be5f6524-6406-46eb-bf85-8e1ca741a24d-kube-api-access-kc2fd\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.090483 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.090422    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-dshm\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.090483 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.090450    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-model-cache\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.090694 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.090499    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-home\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.090835 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.090814    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-tmp-dir\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.090913 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.090833    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-home\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.090970 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.090913    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-model-cache\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.092589 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.092572    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-dshm\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.092883 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.092867    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be5f6524-6406-46eb-bf85-8e1ca741a24d-tls-certs\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.099619 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.099596    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc2fd\" (UniqueName: \"kubernetes.io/projected/be5f6524-6406-46eb-bf85-8e1ca741a24d-kube-api-access-kc2fd\") pod \"test-model-da4b33-kserve-f996b699-nb4nq\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") " pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.191965 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.191939    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:30:52.321565 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.321539    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"]
Jul 30 11:30:52.324033 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:30:52.324006    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe5f6524_6406_46eb_bf85_8e1ca741a24d.slice/crio-8e3fffc2a4d03d2b44ebf9d0fb80d0b9536995bb83694ef7da25778441cb575f WatchSource:0}: Error finding container 8e3fffc2a4d03d2b44ebf9d0fb80d0b9536995bb83694ef7da25778441cb575f: Status 404 returned error can't find the container with id 8e3fffc2a4d03d2b44ebf9d0fb80d0b9536995bb83694ef7da25778441cb575f
Jul 30 11:30:52.589288 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.589193    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq" event={"ID":"be5f6524-6406-46eb-bf85-8e1ca741a24d","Type":"ContainerStarted","Data":"1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352"}
Jul 30 11:30:52.589288 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.589235    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq" event={"ID":"be5f6524-6406-46eb-bf85-8e1ca741a24d","Type":"ContainerStarted","Data":"8e3fffc2a4d03d2b44ebf9d0fb80d0b9536995bb83694ef7da25778441cb575f"}
Jul 30 11:30:52.609890 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:52.609838    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq" podStartSLOduration=1.609820602 podStartE2EDuration="1.609820602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:30:52.608999012 +0000 UTC m=+2558.329883277" watchObservedRunningTime="2026-07-30 11:30:52.609820602 +0000 UTC m=+2558.330704863"
Jul 30 11:30:54.198590 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:30:54.198556    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:31:02.192177 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:02.192141    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:31:02.192667 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:02.192399    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:31:02.204431 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:02.204411    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:31:02.639741 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:02.639666    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:31:03.199427 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:03.199387    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:31:13.404032 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.403997    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"]
Jul 30 11:31:13.407311 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.407294    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.411726 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.411689    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cac774\"/\"test-model-cac774-kserve-self-signed-certs\""
Jul 30 11:31:13.411880 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.411742    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cac774\"/\"default-dockercfg-cd776\""
Jul 30 11:31:13.411880 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.411756    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cac774\"/\"openshift-service-ca.crt\""
Jul 30 11:31:13.412003 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.411972    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cac774\"/\"kube-root-ca.crt\""
Jul 30 11:31:13.421707 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.421686    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"]
Jul 30 11:31:13.454201 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.454177    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:31:13.456423 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.456403    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-home\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.456517 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.456447    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2820377b-41de-4234-af45-ee6c31883d31-tls-certs\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.456517 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.456468    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-tmp-dir\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.456588 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.456518    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-model-cache\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.456588 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.456533    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w622p\" (UniqueName: \"kubernetes.io/projected/2820377b-41de-4234-af45-ee6c31883d31-kube-api-access-w622p\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.456588 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.456575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-dshm\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.557227 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.557197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-model-cache\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.557227 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.557230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w622p\" (UniqueName: \"kubernetes.io/projected/2820377b-41de-4234-af45-ee6c31883d31-kube-api-access-w622p\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.557472 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.557258    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-dshm\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.557472 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.557382    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-home\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.557472 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.557468    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2820377b-41de-4234-af45-ee6c31883d31-tls-certs\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.557646 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.557503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-tmp-dir\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.557714 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.557695    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-model-cache\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.557714 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.557708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-home\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.557878 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.557855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-tmp-dir\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.559484 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.559466    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-dshm\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.559846 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.559828    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2820377b-41de-4234-af45-ee6c31883d31-tls-certs\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.574772 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.574746    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w622p\" (UniqueName: \"kubernetes.io/projected/2820377b-41de-4234-af45-ee6c31883d31-kube-api-access-w622p\") pod \"test-model-cac774-kserve-679fb99c87-clmlc\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") " pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.717165 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.717132    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:13.846764 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:13.846740    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"]
Jul 30 11:31:13.848468 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:31:13.848443    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2820377b_41de_4234_af45_ee6c31883d31.slice/crio-c6a0877c937e76718616476a272cb9793fc7e16757bede020b4a36d53544773b WatchSource:0}: Error finding container c6a0877c937e76718616476a272cb9793fc7e16757bede020b4a36d53544773b: Status 404 returned error can't find the container with id c6a0877c937e76718616476a272cb9793fc7e16757bede020b4a36d53544773b
Jul 30 11:31:14.676479 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:14.676449    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc" event={"ID":"2820377b-41de-4234-af45-ee6c31883d31","Type":"ContainerStarted","Data":"ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c"}
Jul 30 11:31:14.676479 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:14.676485    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc" event={"ID":"2820377b-41de-4234-af45-ee6c31883d31","Type":"ContainerStarted","Data":"c6a0877c937e76718616476a272cb9793fc7e16757bede020b4a36d53544773b"}
Jul 30 11:31:14.695791 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:14.695743    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc" podStartSLOduration=1.6957277849999999 podStartE2EDuration="1.695727785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:31:14.694488543 +0000 UTC m=+2580.415372809" watchObservedRunningTime="2026-07-30 11:31:14.695727785 +0000 UTC m=+2580.416612045"
Jul 30 11:31:16.101251 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:16.101216    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:31:23.717901 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:23.717864    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:23.718413 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:23.717925    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:23.730367 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:23.730343    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:24.728845 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:24.728808    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:31:25.998799 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:25.998764    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:31:54.866890 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:54.866859    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"]
Jul 30 11:31:54.888733 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:54.888707    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:31:55.795450 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:55.795418    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"]
Jul 30 11:31:55.814458 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:55.814427    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:31:58.596173 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:31:58.596131    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:32:06.344479 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:06.344442    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:06.443626 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:06.443595    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:06.528446 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:06.528412    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:06.589004 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:06.588977    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:06.690944 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:06.690912    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:06.753306 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:06.753280    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:07.113349 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:07.113271    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:08.805313 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:08.805277    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:32:09.885272 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:09.885236    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq" podUID="be5f6524-6406-46eb-bf85-8e1ca741a24d" containerName="main" containerID="cri-o://1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352" gracePeriod=45
Jul 30 11:32:10.121671 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.121645    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:32:10.203082 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203057    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-tmp-dir\") pod \"be5f6524-6406-46eb-bf85-8e1ca741a24d\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") "
Jul 30 11:32:10.203269 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203150    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-model-cache\") pod \"be5f6524-6406-46eb-bf85-8e1ca741a24d\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") "
Jul 30 11:32:10.203269 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203209    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc2fd\" (UniqueName: \"kubernetes.io/projected/be5f6524-6406-46eb-bf85-8e1ca741a24d-kube-api-access-kc2fd\") pod \"be5f6524-6406-46eb-bf85-8e1ca741a24d\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") "
Jul 30 11:32:10.203269 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203236    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-home\") pod \"be5f6524-6406-46eb-bf85-8e1ca741a24d\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") "
Jul 30 11:32:10.203269 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203259    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be5f6524-6406-46eb-bf85-8e1ca741a24d-tls-certs\") pod \"be5f6524-6406-46eb-bf85-8e1ca741a24d\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") "
Jul 30 11:32:10.203474 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203311    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-dshm\") pod \"be5f6524-6406-46eb-bf85-8e1ca741a24d\" (UID: \"be5f6524-6406-46eb-bf85-8e1ca741a24d\") "
Jul 30 11:32:10.203474 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203307    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be5f6524-6406-46eb-bf85-8e1ca741a24d" (UID: "be5f6524-6406-46eb-bf85-8e1ca741a24d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:32:10.203572 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203460    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-model-cache" (OuterVolumeSpecName: "model-cache") pod "be5f6524-6406-46eb-bf85-8e1ca741a24d" (UID: "be5f6524-6406-46eb-bf85-8e1ca741a24d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:32:10.203572 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203524    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-home" (OuterVolumeSpecName: "home") pod "be5f6524-6406-46eb-bf85-8e1ca741a24d" (UID: "be5f6524-6406-46eb-bf85-8e1ca741a24d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:32:10.203572 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203538    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:10.203572 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.203555    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:10.205465 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.205444    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-dshm" (OuterVolumeSpecName: "dshm") pod "be5f6524-6406-46eb-bf85-8e1ca741a24d" (UID: "be5f6524-6406-46eb-bf85-8e1ca741a24d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:32:10.205569 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.205498    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5f6524-6406-46eb-bf85-8e1ca741a24d-kube-api-access-kc2fd" (OuterVolumeSpecName: "kube-api-access-kc2fd") pod "be5f6524-6406-46eb-bf85-8e1ca741a24d" (UID: "be5f6524-6406-46eb-bf85-8e1ca741a24d"). InnerVolumeSpecName "kube-api-access-kc2fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:32:10.205569 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.205515    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5f6524-6406-46eb-bf85-8e1ca741a24d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be5f6524-6406-46eb-bf85-8e1ca741a24d" (UID: "be5f6524-6406-46eb-bf85-8e1ca741a24d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:32:10.304073 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.304040    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc2fd\" (UniqueName: \"kubernetes.io/projected/be5f6524-6406-46eb-bf85-8e1ca741a24d-kube-api-access-kc2fd\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:10.304073 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.304069    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:10.304073 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.304079    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be5f6524-6406-46eb-bf85-8e1ca741a24d-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:10.304305 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.304088    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be5f6524-6406-46eb-bf85-8e1ca741a24d-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:10.813753 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.813698    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc" podUID="2820377b-41de-4234-af45-ee6c31883d31" containerName="main" containerID="cri-o://ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c" gracePeriod=45
Jul 30 11:32:10.898945 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.898917    2571 generic.go:356] "Generic (PLEG): container finished" podID="be5f6524-6406-46eb-bf85-8e1ca741a24d" containerID="1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352" exitCode=0
Jul 30 11:32:10.899264 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.898957    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq" event={"ID":"be5f6524-6406-46eb-bf85-8e1ca741a24d","Type":"ContainerDied","Data":"1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352"}
Jul 30 11:32:10.899264 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.898981    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq" event={"ID":"be5f6524-6406-46eb-bf85-8e1ca741a24d","Type":"ContainerDied","Data":"8e3fffc2a4d03d2b44ebf9d0fb80d0b9536995bb83694ef7da25778441cb575f"}
Jul 30 11:32:10.899264 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.898977    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"
Jul 30 11:32:10.899264 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.899001    2571 scope.go:117] "RemoveContainer" containerID="1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352"
Jul 30 11:32:10.915986 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.915965    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"]
Jul 30 11:32:10.921139 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.921117    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-da4b33/test-model-da4b33-kserve-f996b699-nb4nq"]
Jul 30 11:32:10.924169 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.924149    2571 scope.go:117] "RemoveContainer" containerID="1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352"
Jul 30 11:32:10.924482 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:32:10.924465    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352\": container with ID starting with 1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352 not found: ID does not exist" containerID="1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352"
Jul 30 11:32:10.924529 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:10.924488    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352"} err="failed to get container status \"1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352\": rpc error: code = NotFound desc = could not find container \"1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352\": container with ID starting with 1db92e3a6ebfeb89597f7cbb6f35460bc1a685a6e14e7f252ab7377c84aff352 not found: ID does not exist"
Jul 30 11:32:11.052277 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.052253    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:32:11.210044 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.210018    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-home\") pod \"2820377b-41de-4234-af45-ee6c31883d31\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") "
Jul 30 11:32:11.210267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.210081    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-tmp-dir\") pod \"2820377b-41de-4234-af45-ee6c31883d31\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") "
Jul 30 11:32:11.210267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.210135    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-dshm\") pod \"2820377b-41de-4234-af45-ee6c31883d31\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") "
Jul 30 11:32:11.210267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.210170    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w622p\" (UniqueName: \"kubernetes.io/projected/2820377b-41de-4234-af45-ee6c31883d31-kube-api-access-w622p\") pod \"2820377b-41de-4234-af45-ee6c31883d31\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") "
Jul 30 11:32:11.210267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.210197    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2820377b-41de-4234-af45-ee6c31883d31-tls-certs\") pod \"2820377b-41de-4234-af45-ee6c31883d31\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") "
Jul 30 11:32:11.210267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.210226    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-model-cache\") pod \"2820377b-41de-4234-af45-ee6c31883d31\" (UID: \"2820377b-41de-4234-af45-ee6c31883d31\") "
Jul 30 11:32:11.210506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.210361    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2820377b-41de-4234-af45-ee6c31883d31" (UID: "2820377b-41de-4234-af45-ee6c31883d31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:32:11.210506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.210377    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-home" (OuterVolumeSpecName: "home") pod "2820377b-41de-4234-af45-ee6c31883d31" (UID: "2820377b-41de-4234-af45-ee6c31883d31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:32:11.210626 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.210543    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-model-cache" (OuterVolumeSpecName: "model-cache") pod "2820377b-41de-4234-af45-ee6c31883d31" (UID: "2820377b-41de-4234-af45-ee6c31883d31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:32:11.212464 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.212439    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2820377b-41de-4234-af45-ee6c31883d31-kube-api-access-w622p" (OuterVolumeSpecName: "kube-api-access-w622p") pod "2820377b-41de-4234-af45-ee6c31883d31" (UID: "2820377b-41de-4234-af45-ee6c31883d31"). InnerVolumeSpecName "kube-api-access-w622p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:32:11.212564 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.212468    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2820377b-41de-4234-af45-ee6c31883d31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2820377b-41de-4234-af45-ee6c31883d31" (UID: "2820377b-41de-4234-af45-ee6c31883d31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:32:11.212564 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.212498    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-dshm" (OuterVolumeSpecName: "dshm") pod "2820377b-41de-4234-af45-ee6c31883d31" (UID: "2820377b-41de-4234-af45-ee6c31883d31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:32:11.311567 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.311536    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:11.311567 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.311560    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:11.311758 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.311587    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:11.311758 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.311598    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2820377b-41de-4234-af45-ee6c31883d31-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:11.311758 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.311611    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w622p\" (UniqueName: \"kubernetes.io/projected/2820377b-41de-4234-af45-ee6c31883d31-kube-api-access-w622p\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:11.311758 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.311621    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2820377b-41de-4234-af45-ee6c31883d31-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:32:11.905020 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.904984    2571 generic.go:356] "Generic (PLEG): container finished" podID="2820377b-41de-4234-af45-ee6c31883d31" containerID="ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c" exitCode=0
Jul 30 11:32:11.905618 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.905038    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"
Jul 30 11:32:11.905618 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.905071    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc" event={"ID":"2820377b-41de-4234-af45-ee6c31883d31","Type":"ContainerDied","Data":"ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c"}
Jul 30 11:32:11.905618 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.905132    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc" event={"ID":"2820377b-41de-4234-af45-ee6c31883d31","Type":"ContainerDied","Data":"c6a0877c937e76718616476a272cb9793fc7e16757bede020b4a36d53544773b"}
Jul 30 11:32:11.905618 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.905157    2571 scope.go:117] "RemoveContainer" containerID="ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c"
Jul 30 11:32:11.913822 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.913806    2571 scope.go:117] "RemoveContainer" containerID="ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c"
Jul 30 11:32:11.914072 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:32:11.914052    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c\": container with ID starting with ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c not found: ID does not exist" containerID="ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c"
Jul 30 11:32:11.914142 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.914080    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c"} err="failed to get container status \"ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c\": rpc error: code = NotFound desc = could not find container \"ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c\": container with ID starting with ae640d5ae9e15b3d40a3d965fe1e58dfff72d04cf5ad738a6706fed3da427b8c not found: ID does not exist"
Jul 30 11:32:11.927446 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.927421    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"]
Jul 30 11:32:11.931029 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:11.931006    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cac774/test-model-cac774-kserve-679fb99c87-clmlc"]
Jul 30 11:32:12.854113 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:12.854058    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2820377b-41de-4234-af45-ee6c31883d31" path="/var/lib/kubelet/pods/2820377b-41de-4234-af45-ee6c31883d31/volumes"
Jul 30 11:32:12.854549 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:12.854527    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be5f6524-6406-46eb-bf85-8e1ca741a24d" path="/var/lib/kubelet/pods/be5f6524-6406-46eb-bf85-8e1ca741a24d/volumes"
Jul 30 11:32:13.358653 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:13.358624    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:13.453807 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:13.453778    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:13.641415 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:13.641333    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:13.878002 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:13.877967    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:24.548056 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:24.548025    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:24.651467 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:24.651436    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:24.753048 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:24.753016    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:24.869919 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:24.869841    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:25.256313 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:25.256280    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:26.691682 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:26.691647    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:32:31.441943 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:31.441909    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:31.543784 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:31.543751    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:31.664073 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:31.664025    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:31.899234 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:31.899152    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:32.240355 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:32.240324    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:32.574290 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:32.574203    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:32.827932 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:32.827839    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:36.090679 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:36.090641    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:36.193546 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:36.193516    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:37.429033 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:37.428997    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:37.894653 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:37.894565    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:38.245507 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:38.245479    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:38.376768 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:38.376739    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:32:40.901205 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:32:40.901172    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:33:01.006871 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:01.006842    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:01.507433 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:01.507401    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:01.800230 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:01.800152    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:02.056981 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:02.056880    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:04.999067 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:04.999033    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:05.093532 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:05.093499    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:06.148327 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:06.148296    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:06.469778 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:06.469746    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:06.657204 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:06.657173    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:06.758598 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:06.758524    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:08.490555 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:08.490513    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:33:14.402175 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:14.402142    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:27.004574 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:27.004514    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:29.997175 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:29.997143    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:31.660217 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:31.660172    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:31.752384 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:31.752349    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:32.940324 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.940293    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"]
Jul 30 11:33:32.940764 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.940541    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2820377b-41de-4234-af45-ee6c31883d31" containerName="main"
Jul 30 11:33:32.940764 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.940551    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2820377b-41de-4234-af45-ee6c31883d31" containerName="main"
Jul 30 11:33:32.940764 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.940561    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be5f6524-6406-46eb-bf85-8e1ca741a24d" containerName="main"
Jul 30 11:33:32.940764 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.940567    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="be5f6524-6406-46eb-bf85-8e1ca741a24d" containerName="main"
Jul 30 11:33:32.940764 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.940621    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="be5f6524-6406-46eb-bf85-8e1ca741a24d" containerName="main"
Jul 30 11:33:32.940764 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.940628    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2820377b-41de-4234-af45-ee6c31883d31" containerName="main"
Jul 30 11:33:32.943413 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.943397    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"
Jul 30 11:33:32.946784 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.946762    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\""
Jul 30 11:33:32.946882 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.946774    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 11:33:32.960321 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:32.960300    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"]
Jul 30 11:33:33.019641 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:33.019615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e2e5241-ea03-4d50-8468-2610578ece5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2\" (UID: \"3e2e5241-ea03-4d50-8468-2610578ece5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"
Jul 30 11:33:33.120839 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:33.120798    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e2e5241-ea03-4d50-8468-2610578ece5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2\" (UID: \"3e2e5241-ea03-4d50-8468-2610578ece5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"
Jul 30 11:33:33.121459 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:33.121438    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e2e5241-ea03-4d50-8468-2610578ece5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2\" (UID: \"3e2e5241-ea03-4d50-8468-2610578ece5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"
Jul 30 11:33:33.252797 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:33.252715    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"
Jul 30 11:33:33.380056 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:33.380007    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"]
Jul 30 11:33:33.382281 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:33:33.382251    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2e5241_ea03_4d50_8468_2610578ece5c.slice/crio-0e94b25ea134cd2ba7f8beb85a6566e0edb273c605bc58b1751b98a46f0a3100 WatchSource:0}: Error finding container 0e94b25ea134cd2ba7f8beb85a6566e0edb273c605bc58b1751b98a46f0a3100: Status 404 returned error can't find the container with id 0e94b25ea134cd2ba7f8beb85a6566e0edb273c605bc58b1751b98a46f0a3100
Jul 30 11:33:34.231450 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:34.231416    2571 generic.go:356] "Generic (PLEG): container finished" podID="3e2e5241-ea03-4d50-8468-2610578ece5c" containerID="d6e70747033d1385c01aaa21d439eda8dda44906e237d23dd401e7d8a8590baf" exitCode=0
Jul 30 11:33:34.231840 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:34.231462    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2" event={"ID":"3e2e5241-ea03-4d50-8468-2610578ece5c","Type":"ContainerDied","Data":"d6e70747033d1385c01aaa21d439eda8dda44906e237d23dd401e7d8a8590baf"}
Jul 30 11:33:34.231840 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:34.231525    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2" event={"ID":"3e2e5241-ea03-4d50-8468-2610578ece5c","Type":"ContainerStarted","Data":"0e94b25ea134cd2ba7f8beb85a6566e0edb273c605bc58b1751b98a46f0a3100"}
Jul 30 11:33:35.362441 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:35.362416    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"
Jul 30 11:33:35.438248 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:35.438224    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e2e5241-ea03-4d50-8468-2610578ece5c-maas-api-service-ca\") pod \"3e2e5241-ea03-4d50-8468-2610578ece5c\" (UID: \"3e2e5241-ea03-4d50-8468-2610578ece5c\") "
Jul 30 11:33:35.438597 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:35.438573    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2e5241-ea03-4d50-8468-2610578ece5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e2e5241-ea03-4d50-8468-2610578ece5c" (UID: "3e2e5241-ea03-4d50-8468-2610578ece5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:33:35.539806 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:35.539736    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e2e5241-ea03-4d50-8468-2610578ece5c-maas-api-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:33:36.240463 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:36.240438    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"
Jul 30 11:33:36.240643 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:36.240437    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2" event={"ID":"3e2e5241-ea03-4d50-8468-2610578ece5c","Type":"ContainerDied","Data":"0e94b25ea134cd2ba7f8beb85a6566e0edb273c605bc58b1751b98a46f0a3100"}
Jul 30 11:33:36.240643 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:36.240564    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e94b25ea134cd2ba7f8beb85a6566e0edb273c605bc58b1751b98a46f0a3100"
Jul 30 11:33:48.851734 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:48.851676    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:49.072816 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:49.072783    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"]
Jul 30 11:33:49.075731 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:49.075708    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d6bcbf-v4zp2"]
Jul 30 11:33:50.407201 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:50.407169    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:50.854323 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:50.854290    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e2e5241-ea03-4d50-8468-2610578ece5c" path="/var/lib/kubelet/pods/3e2e5241-ea03-4d50-8468-2610578ece5c/volumes"
Jul 30 11:33:51.100961 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:51.100922    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:33:55.493718 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:55.493653    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:55.557553 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:55.557517    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:55.759305 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:55.759223    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:55.962034 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:55.962006    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:33:56.082141 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.082035    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"]
Jul 30 11:33:56.082349 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.082337    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2e5241-ea03-4d50-8468-2610578ece5c" containerName="revoke-keys"
Jul 30 11:33:56.082400 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.082351    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2e5241-ea03-4d50-8468-2610578ece5c" containerName="revoke-keys"
Jul 30 11:33:56.082439 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.082407    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2e5241-ea03-4d50-8468-2610578ece5c" containerName="revoke-keys"
Jul 30 11:33:56.085074 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.085057    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"
Jul 30 11:33:56.088049 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.088028    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\""
Jul 30 11:33:56.088049 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.088047    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 11:33:56.094144 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.094121    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"]
Jul 30 11:33:56.198348 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.198312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/224fdf4f-e3ca-42b1-bb88-5332b5217817-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7\" (UID: \"224fdf4f-e3ca-42b1-bb88-5332b5217817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"
Jul 30 11:33:56.298671 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.298634    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/224fdf4f-e3ca-42b1-bb88-5332b5217817-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7\" (UID: \"224fdf4f-e3ca-42b1-bb88-5332b5217817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"
Jul 30 11:33:56.299357 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.299337    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/224fdf4f-e3ca-42b1-bb88-5332b5217817-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7\" (UID: \"224fdf4f-e3ca-42b1-bb88-5332b5217817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"
Jul 30 11:33:56.395859 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.395748    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"
Jul 30 11:33:56.524629 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:56.524605    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"]
Jul 30 11:33:56.527217 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:33:56.527192    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224fdf4f_e3ca_42b1_bb88_5332b5217817.slice/crio-f19704334088623c32e8785c9775a1ff9df9e3d30bdcd55616c739e0c53647ca WatchSource:0}: Error finding container f19704334088623c32e8785c9775a1ff9df9e3d30bdcd55616c739e0c53647ca: Status 404 returned error can't find the container with id f19704334088623c32e8785c9775a1ff9df9e3d30bdcd55616c739e0c53647ca
Jul 30 11:33:57.322456 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:57.322418    2571 generic.go:356] "Generic (PLEG): container finished" podID="224fdf4f-e3ca-42b1-bb88-5332b5217817" containerID="4e12ce0406eefcc1dc1bacae70ba8e9336cbe9c277692cf36c5a79a6c4f5c97b" exitCode=0
Jul 30 11:33:57.322611 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:57.322466    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7" event={"ID":"224fdf4f-e3ca-42b1-bb88-5332b5217817","Type":"ContainerDied","Data":"4e12ce0406eefcc1dc1bacae70ba8e9336cbe9c277692cf36c5a79a6c4f5c97b"}
Jul 30 11:33:57.322611 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:57.322517    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7" event={"ID":"224fdf4f-e3ca-42b1-bb88-5332b5217817","Type":"ContainerStarted","Data":"f19704334088623c32e8785c9775a1ff9df9e3d30bdcd55616c739e0c53647ca"}
Jul 30 11:33:58.453602 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:58.453573    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"
Jul 30 11:33:58.617647 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:58.617545    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/224fdf4f-e3ca-42b1-bb88-5332b5217817-maas-api-service-ca\") pod \"224fdf4f-e3ca-42b1-bb88-5332b5217817\" (UID: \"224fdf4f-e3ca-42b1-bb88-5332b5217817\") "
Jul 30 11:33:58.617904 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:58.617880    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224fdf4f-e3ca-42b1-bb88-5332b5217817-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "224fdf4f-e3ca-42b1-bb88-5332b5217817" (UID: "224fdf4f-e3ca-42b1-bb88-5332b5217817"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:33:58.718054 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:58.718019    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/224fdf4f-e3ca-42b1-bb88-5332b5217817-maas-api-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:33:59.331689 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:59.331654    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7" event={"ID":"224fdf4f-e3ca-42b1-bb88-5332b5217817","Type":"ContainerDied","Data":"f19704334088623c32e8785c9775a1ff9df9e3d30bdcd55616c739e0c53647ca"}
Jul 30 11:33:59.331689 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:59.331689    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"
Jul 30 11:33:59.333815 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:33:59.331712    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19704334088623c32e8785c9775a1ff9df9e3d30bdcd55616c739e0c53647ca"
Jul 30 11:34:11.721109 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:11.721054    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:11.779346 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:11.779311    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:11.880424 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:11.880388    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:11.890912 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:11.890878    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"]
Jul 30 11:34:11.893696 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:11.893672    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c382f-5tjj7"]
Jul 30 11:34:11.980133 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:11.980040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:12.285742 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:12.285660    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:12.857360 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:12.857324    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="224fdf4f-e3ca-42b1-bb88-5332b5217817" path="/var/lib/kubelet/pods/224fdf4f-e3ca-42b1-bb88-5332b5217817/volumes"
Jul 30 11:34:13.790448 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:13.790409    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:34:18.232415 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:18.232386    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:18.296111 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:18.296079    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:18.513657 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:18.513541    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:18.763283 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:18.763249    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:19.098949 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:19.098909    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:19.466330 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:19.466295    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:19.817043 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:19.816956    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:22.751734 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:22.751697    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:22.852000 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:22.851974    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:23.883024 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:23.882982    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:24.019382 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:24.019354    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:24.505123 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:24.505072    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:24.611320 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:24.611288    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:27.097569 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:27.097539    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:34:47.749294 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:47.749222    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:48.118351 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:48.118253    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:48.450229 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:48.450203    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:48.645049 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:48.645019    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:51.654538 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:51.654504    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:51.744870 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:51.744835    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:52.918372 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:52.918327    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:53.233020 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.232982    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:53.587056 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.586979    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:53.688221 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.688182    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:34:53.723248 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.723216    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"]
Jul 30 11:34:53.723479 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.723467    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224fdf4f-e3ca-42b1-bb88-5332b5217817" containerName="revoke-keys"
Jul 30 11:34:53.723525 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.723480    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="224fdf4f-e3ca-42b1-bb88-5332b5217817" containerName="revoke-keys"
Jul 30 11:34:53.723560 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.723547    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="224fdf4f-e3ca-42b1-bb88-5332b5217817" containerName="revoke-keys"
Jul 30 11:34:53.726292 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.726276    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:34:53.731192 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.731167    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-945db9\""
Jul 30 11:34:53.731324 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.731218    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-945db9-dockercfg-zfvtx\""
Jul 30 11:34:53.750194 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.750163    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"]
Jul 30 11:34:53.804855 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.804826    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4cvw\" (UniqueName: \"kubernetes.io/projected/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-kube-api-access-b4cvw\") pod \"payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz\" (UID: \"0af58b7e-d20d-47bf-a07c-b9c43ca312f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:34:53.804975 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.804878    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz\" (UID: \"0af58b7e-d20d-47bf-a07c-b9c43ca312f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:34:53.905271 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.905190    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4cvw\" (UniqueName: \"kubernetes.io/projected/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-kube-api-access-b4cvw\") pod \"payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz\" (UID: \"0af58b7e-d20d-47bf-a07c-b9c43ca312f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:34:53.905271 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.905248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz\" (UID: \"0af58b7e-d20d-47bf-a07c-b9c43ca312f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:34:53.905955 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.905937    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz\" (UID: \"0af58b7e-d20d-47bf-a07c-b9c43ca312f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:34:53.914125 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:53.914085    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4cvw\" (UniqueName: \"kubernetes.io/projected/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-kube-api-access-b4cvw\") pod \"payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz\" (UID: \"0af58b7e-d20d-47bf-a07c-b9c43ca312f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:34:54.036416 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:54.036381    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:34:54.169297 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:54.169220    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"]
Jul 30 11:34:54.172863 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:34:54.172826    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af58b7e_d20d_47bf_a07c_b9c43ca312f2.slice/crio-e5516faf4c07c53ec0ab93c8ff6564b6e968a0046515eb5c971aa3b050432efa WatchSource:0}: Error finding container e5516faf4c07c53ec0ab93c8ff6564b6e968a0046515eb5c971aa3b050432efa: Status 404 returned error can't find the container with id e5516faf4c07c53ec0ab93c8ff6564b6e968a0046515eb5c971aa3b050432efa
Jul 30 11:34:54.174759 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:54.174741    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 11:34:54.549977 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:54.549939    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz" event={"ID":"0af58b7e-d20d-47bf-a07c-b9c43ca312f2","Type":"ContainerStarted","Data":"e5516faf4c07c53ec0ab93c8ff6564b6e968a0046515eb5c971aa3b050432efa"}
Jul 30 11:34:55.808225 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:55.808193    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:34:56.561962 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:56.561920    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz" event={"ID":"0af58b7e-d20d-47bf-a07c-b9c43ca312f2","Type":"ContainerStarted","Data":"d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d"}
Jul 30 11:34:56.562178 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:56.562050    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:34:56.583164 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:34:56.583119    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz" podStartSLOduration=1.7471135979999999 podStartE2EDuration="3.583088s" totalImagesPullingTime="1.835974402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:34:56.582150816 +0000 UTC m=+2802.303035077" watchObservedRunningTime="2026-07-30 11:34:56.583088 +0000 UTC m=+2802.303972260"
Jul 30 11:35:07.568293 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:07.568262    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:35:16.650845 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:16.650811    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:35:28.388926 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.388890    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"]
Jul 30 11:35:28.391861 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.391845    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.394877 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.394844    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-060e9f\"/\"default-dockercfg-xm2jv\""
Jul 30 11:35:28.395019 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.394921    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-060e9f\"/\"kube-root-ca.crt\""
Jul 30 11:35:28.396073 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.396055    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-060e9f\"/\"openshift-service-ca.crt\""
Jul 30 11:35:28.396386 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.396364    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-060e9f\"/\"auth-test-model-060e9f-kserve-self-signed-certs\""
Jul 30 11:35:28.404641 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.404615    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"]
Jul 30 11:35:28.413215 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.413191    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:35:28.470442 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.470389    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-home\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.470442 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.470445    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-dshm\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.470679 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.470494    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6901f37e-08bf-4acd-a1ea-213180056f26-tls-certs\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.470679 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.470561    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-model-cache\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.470679 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.470602    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8trx\" (UniqueName: \"kubernetes.io/projected/6901f37e-08bf-4acd-a1ea-213180056f26-kube-api-access-w8trx\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.470679 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.470665    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-tmp-dir\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.571775 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.571741    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8trx\" (UniqueName: \"kubernetes.io/projected/6901f37e-08bf-4acd-a1ea-213180056f26-kube-api-access-w8trx\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.571942 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.571794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-tmp-dir\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.571942 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.571818    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-home\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.571942 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.571836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-dshm\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.571942 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.571856    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6901f37e-08bf-4acd-a1ea-213180056f26-tls-certs\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.571942 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.571885    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-model-cache\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.572398 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.572373    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-home\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.572473 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.572403    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-model-cache\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.572473 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.572437    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-tmp-dir\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.574135 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.574113    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-dshm\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.574439 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.574418    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6901f37e-08bf-4acd-a1ea-213180056f26-tls-certs\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.581880 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.581851    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8trx\" (UniqueName: \"kubernetes.io/projected/6901f37e-08bf-4acd-a1ea-213180056f26-kube-api-access-w8trx\") pod \"auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") " pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.701199 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.701168    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:28.829402 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:28.829375    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"]
Jul 30 11:35:28.831670 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:35:28.831643    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6901f37e_08bf_4acd_a1ea_213180056f26.slice/crio-4d11580001aedd5e061f4860631489ef77f0683455bd0f3341e3aee42cc71ea2 WatchSource:0}: Error finding container 4d11580001aedd5e061f4860631489ef77f0683455bd0f3341e3aee42cc71ea2: Status 404 returned error can't find the container with id 4d11580001aedd5e061f4860631489ef77f0683455bd0f3341e3aee42cc71ea2
Jul 30 11:35:29.700441 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:29.700404    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p" event={"ID":"6901f37e-08bf-4acd-a1ea-213180056f26","Type":"ContainerStarted","Data":"ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e"}
Jul 30 11:35:29.700834 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:29.700448    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p" event={"ID":"6901f37e-08bf-4acd-a1ea-213180056f26","Type":"ContainerStarted","Data":"4d11580001aedd5e061f4860631489ef77f0683455bd0f3341e3aee42cc71ea2"}
Jul 30 11:35:29.722860 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:29.722809    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p" podStartSLOduration=1.722794125 podStartE2EDuration="1.722794125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:35:29.722235221 +0000 UTC m=+2835.443119481" watchObservedRunningTime="2026-07-30 11:35:29.722794125 +0000 UTC m=+2835.443678385"
Jul 30 11:35:30.801978 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:30.801942    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:35:38.701848 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:38.701810    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:38.702248 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:38.701866    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:38.714038 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:38.714019    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:38.751048 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:38.751021    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:35:46.154842 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:46.154812    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:35:58.814874 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.814838    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"]
Jul 30 11:35:58.817925 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.817908    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.821484 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.821458    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"auth-test-model-945db9-kserve-self-signed-certs\""
Jul 30 11:35:58.822534 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.822510    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"openshift-service-ca.crt\""
Jul 30 11:35:58.822652 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.822622    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"default-dockercfg-l62n8\""
Jul 30 11:35:58.822746 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.822726    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"kube-root-ca.crt\""
Jul 30 11:35:58.830712 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.830689    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"]
Jul 30 11:35:58.890438 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.890396    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-home\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.890438 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.890443    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-model-cache\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.890687 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.890491    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tls-certs\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.890687 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.890542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jds9\" (UniqueName: \"kubernetes.io/projected/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-kube-api-access-7jds9\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.890687 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.890620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tmp-dir\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.890687 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.890652    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-dshm\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.945318 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.945287    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:35:58.991022 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.990985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-model-cache\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.991242 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.991026    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tls-certs\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.991242 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.991071    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jds9\" (UniqueName: \"kubernetes.io/projected/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-kube-api-access-7jds9\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.991242 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.991162    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tmp-dir\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.991242 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.991187    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-dshm\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.991475 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.991321    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-home\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.991475 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.991383    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-model-cache\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.991563 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.991520    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tmp-dir\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.991639 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.991620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-home\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.993299 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.993278    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-dshm\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:58.993640 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:58.993624    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tls-certs\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:59.001266 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:59.001240    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jds9\" (UniqueName: \"kubernetes.io/projected/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-kube-api-access-7jds9\") pod \"auth-test-model-945db9-kserve-66878fd66c-jllnq\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") " pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:59.128038 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:59.127951    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:35:59.258130 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:59.258090    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"]
Jul 30 11:35:59.260755 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:35:59.260727    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6eb0a0d_5d9f_4c16_bec2_23421c47514b.slice/crio-e70932face29861d4f1bfcdd2ce1758623ef481b4a38eca8f3ab788729ea466e WatchSource:0}: Error finding container e70932face29861d4f1bfcdd2ce1758623ef481b4a38eca8f3ab788729ea466e: Status 404 returned error can't find the container with id e70932face29861d4f1bfcdd2ce1758623ef481b4a38eca8f3ab788729ea466e
Jul 30 11:35:59.828868 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:59.828833    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq" event={"ID":"a6eb0a0d-5d9f-4c16-bec2-23421c47514b","Type":"ContainerStarted","Data":"7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51"}
Jul 30 11:35:59.828868 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:59.828874    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq" event={"ID":"a6eb0a0d-5d9f-4c16-bec2-23421c47514b","Type":"ContainerStarted","Data":"e70932face29861d4f1bfcdd2ce1758623ef481b4a38eca8f3ab788729ea466e"}
Jul 30 11:35:59.847855 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:35:59.847809    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq" podStartSLOduration=1.847792225 podStartE2EDuration="1.847792225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:35:59.847784364 +0000 UTC m=+2865.568668620" watchObservedRunningTime="2026-07-30 11:35:59.847792225 +0000 UTC m=+2865.568676491"
Jul 30 11:36:01.400586 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:36:01.400552    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:36:09.128590 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:36:09.128478    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:36:09.128590 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:36:09.128551    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:36:09.141172 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:36:09.141142    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:36:09.879822 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:36:09.879791    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:36:27.495182 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:36:27.495145    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:36:40.731437 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:36:40.731399    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:36:42.890501 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:36:42.890469    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:36:50.711219 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:36:50.711184    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:36:59.403666 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:36:59.403629    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:37:11.300137 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:37:11.300087    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:37:21.594596 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:37:21.594550    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:37:36.093481 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:37:36.093444    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:37:47.486405 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:37:47.486364    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:38:02.797341 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:38:02.797303    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:38:13.893211 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:38:13.893172    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:38:29.188706 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:38:29.188664    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:38:41.083372 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:38:41.083335    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:38:55.808728 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:38:55.808695    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:38:59.883246 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:38:59.883210    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"]
Jul 30 11:38:59.932479 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:38:59.932453    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:39:00.801686 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:00.801650    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"]
Jul 30 11:39:01.826876 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:01.826839    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:39:06.596388 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:06.596309    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:39:12.617214 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.617184    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:39:12.621091 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.621064    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"]
Jul 30 11:39:12.624111 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.624085    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.626845 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.626821    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-060e9f\"/\"sub-test-model-060e9f-kserve-self-signed-certs\""
Jul 30 11:39:12.635951 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.635928    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"]
Jul 30 11:39:12.746536 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.746498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-dshm\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.746715 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.746548    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-model-cache\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.746715 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.746583    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t79z\" (UniqueName: \"kubernetes.io/projected/11906152-d87c-4245-ab2b-f784859ae101-kube-api-access-4t79z\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.746715 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.746615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11906152-d87c-4245-ab2b-f784859ae101-tls-certs\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.746715 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.746642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-tmp-dir\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.746879 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.746714    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-home\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.847372 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.847337    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-dshm\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.847529 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.847380    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-model-cache\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.847529 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.847407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t79z\" (UniqueName: \"kubernetes.io/projected/11906152-d87c-4245-ab2b-f784859ae101-kube-api-access-4t79z\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.847529 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.847447    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11906152-d87c-4245-ab2b-f784859ae101-tls-certs\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.847529 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.847479    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-tmp-dir\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.847529 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.847512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-home\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.847866 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.847841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-model-cache\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.847934 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.847869    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-home\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.847972 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.847933    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-tmp-dir\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.849582 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.849562    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-dshm\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.849980 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.849959    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11906152-d87c-4245-ab2b-f784859ae101-tls-certs\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.862310 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.862283    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t79z\" (UniqueName: \"kubernetes.io/projected/11906152-d87c-4245-ab2b-f784859ae101-kube-api-access-4t79z\") pod \"sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") " pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:12.932534 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:12.932507    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:13.057368 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:13.057336    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"]
Jul 30 11:39:13.060667 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:39:13.060637    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11906152_d87c_4245_ab2b_f784859ae101.slice/crio-0ff17431646a3038712fa1cb62ed7f1e0ecee115f85a936c66baa96b0c035917 WatchSource:0}: Error finding container 0ff17431646a3038712fa1cb62ed7f1e0ecee115f85a936c66baa96b0c035917: Status 404 returned error can't find the container with id 0ff17431646a3038712fa1cb62ed7f1e0ecee115f85a936c66baa96b0c035917
Jul 30 11:39:13.630410 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:13.630373    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47" event={"ID":"11906152-d87c-4245-ab2b-f784859ae101","Type":"ContainerStarted","Data":"993f994ffa23824fdbe13bb1ab44027544bb4563da3f67f0ce25441f5752ca2e"}
Jul 30 11:39:13.630831 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:13.630416    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47" event={"ID":"11906152-d87c-4245-ab2b-f784859ae101","Type":"ContainerStarted","Data":"0ff17431646a3038712fa1cb62ed7f1e0ecee115f85a936c66baa96b0c035917"}
Jul 30 11:39:13.650069 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:13.650022    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47" podStartSLOduration=1.650008084 podStartE2EDuration="1.650008084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:39:13.648731707 +0000 UTC m=+3059.369615971" watchObservedRunningTime="2026-07-30 11:39:13.650008084 +0000 UTC m=+3059.370892377"
Jul 30 11:39:14.901565 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:14.901524    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p" podUID="6901f37e-08bf-4acd-a1ea-213180056f26" containerName="main" containerID="cri-o://ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e" gracePeriod=45
Jul 30 11:39:15.139228 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.139203    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:39:15.157050 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.156991    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:39:15.265293 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.265255    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6901f37e-08bf-4acd-a1ea-213180056f26-tls-certs\") pod \"6901f37e-08bf-4acd-a1ea-213180056f26\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") "
Jul 30 11:39:15.265474 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.265316    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-dshm\") pod \"6901f37e-08bf-4acd-a1ea-213180056f26\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") "
Jul 30 11:39:15.265474 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.265346    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8trx\" (UniqueName: \"kubernetes.io/projected/6901f37e-08bf-4acd-a1ea-213180056f26-kube-api-access-w8trx\") pod \"6901f37e-08bf-4acd-a1ea-213180056f26\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") "
Jul 30 11:39:15.265474 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.265378    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-tmp-dir\") pod \"6901f37e-08bf-4acd-a1ea-213180056f26\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") "
Jul 30 11:39:15.265636 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.265505    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-model-cache\") pod \"6901f37e-08bf-4acd-a1ea-213180056f26\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") "
Jul 30 11:39:15.265636 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.265614    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-home\") pod \"6901f37e-08bf-4acd-a1ea-213180056f26\" (UID: \"6901f37e-08bf-4acd-a1ea-213180056f26\") "
Jul 30 11:39:15.265748 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.265651    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6901f37e-08bf-4acd-a1ea-213180056f26" (UID: "6901f37e-08bf-4acd-a1ea-213180056f26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:39:15.265810 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.265781    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-model-cache" (OuterVolumeSpecName: "model-cache") pod "6901f37e-08bf-4acd-a1ea-213180056f26" (UID: "6901f37e-08bf-4acd-a1ea-213180056f26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:39:15.265971 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.265920    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-home" (OuterVolumeSpecName: "home") pod "6901f37e-08bf-4acd-a1ea-213180056f26" (UID: "6901f37e-08bf-4acd-a1ea-213180056f26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:39:15.266049 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.265980    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:15.266049 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.266001    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:15.267606 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.267581    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6901f37e-08bf-4acd-a1ea-213180056f26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6901f37e-08bf-4acd-a1ea-213180056f26" (UID: "6901f37e-08bf-4acd-a1ea-213180056f26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:39:15.267836 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.267817    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-dshm" (OuterVolumeSpecName: "dshm") pod "6901f37e-08bf-4acd-a1ea-213180056f26" (UID: "6901f37e-08bf-4acd-a1ea-213180056f26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:39:15.267836 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.267828    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6901f37e-08bf-4acd-a1ea-213180056f26-kube-api-access-w8trx" (OuterVolumeSpecName: "kube-api-access-w8trx") pod "6901f37e-08bf-4acd-a1ea-213180056f26" (UID: "6901f37e-08bf-4acd-a1ea-213180056f26"). InnerVolumeSpecName "kube-api-access-w8trx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:39:15.366991 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.366965    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:15.366991 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.366990    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6901f37e-08bf-4acd-a1ea-213180056f26-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:15.367174 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.367003    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6901f37e-08bf-4acd-a1ea-213180056f26-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:15.367174 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.367015    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8trx\" (UniqueName: \"kubernetes.io/projected/6901f37e-08bf-4acd-a1ea-213180056f26-kube-api-access-w8trx\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:15.639835 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.639799    2571 generic.go:356] "Generic (PLEG): container finished" podID="6901f37e-08bf-4acd-a1ea-213180056f26" containerID="ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e" exitCode=0
Jul 30 11:39:15.640010 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.639844    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"
Jul 30 11:39:15.640010 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.639875    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p" event={"ID":"6901f37e-08bf-4acd-a1ea-213180056f26","Type":"ContainerDied","Data":"ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e"}
Jul 30 11:39:15.640010 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.639918    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p" event={"ID":"6901f37e-08bf-4acd-a1ea-213180056f26","Type":"ContainerDied","Data":"4d11580001aedd5e061f4860631489ef77f0683455bd0f3341e3aee42cc71ea2"}
Jul 30 11:39:15.640010 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.639938    2571 scope.go:117] "RemoveContainer" containerID="ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e"
Jul 30 11:39:15.648900 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.648879    2571 scope.go:117] "RemoveContainer" containerID="ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e"
Jul 30 11:39:15.649180 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:39:15.649154    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e\": container with ID starting with ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e not found: ID does not exist" containerID="ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e"
Jul 30 11:39:15.649245 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.649194    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e"} err="failed to get container status \"ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e\": rpc error: code = NotFound desc = could not find container \"ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e\": container with ID starting with ea58af505a5e374a65ca3827f268e1014e5c3e5265e09efa61edde17075f2d8e not found: ID does not exist"
Jul 30 11:39:15.662428 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.662403    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"]
Jul 30 11:39:15.668437 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.668412    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/auth-test-model-060e9f-kserve-675bfbfd7f-kpj2p"]
Jul 30 11:39:15.820048 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:15.820009    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq" podUID="a6eb0a0d-5d9f-4c16-bec2-23421c47514b" containerName="main" containerID="cri-o://7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51" gracePeriod=45
Jul 30 11:39:16.068719 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.068693    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:39:16.174016 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.173928    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-dshm\") pod \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") "
Jul 30 11:39:16.174016 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.173983    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-home\") pod \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") "
Jul 30 11:39:16.174276 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.174052    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tmp-dir\") pod \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") "
Jul 30 11:39:16.174276 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.174088    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tls-certs\") pod \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") "
Jul 30 11:39:16.174276 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.174135    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jds9\" (UniqueName: \"kubernetes.io/projected/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-kube-api-access-7jds9\") pod \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") "
Jul 30 11:39:16.174276 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.174180    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-model-cache\") pod \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\" (UID: \"a6eb0a0d-5d9f-4c16-bec2-23421c47514b\") "
Jul 30 11:39:16.174447 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.174287    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6eb0a0d-5d9f-4c16-bec2-23421c47514b" (UID: "a6eb0a0d-5d9f-4c16-bec2-23421c47514b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:39:16.174447 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.174303    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-home" (OuterVolumeSpecName: "home") pod "a6eb0a0d-5d9f-4c16-bec2-23421c47514b" (UID: "a6eb0a0d-5d9f-4c16-bec2-23421c47514b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:39:16.174557 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.174444    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:16.174557 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.174464    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:16.174557 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.174492    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6eb0a0d-5d9f-4c16-bec2-23421c47514b" (UID: "a6eb0a0d-5d9f-4c16-bec2-23421c47514b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:39:16.176259 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.176233    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-dshm" (OuterVolumeSpecName: "dshm") pod "a6eb0a0d-5d9f-4c16-bec2-23421c47514b" (UID: "a6eb0a0d-5d9f-4c16-bec2-23421c47514b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:39:16.176373 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.176256    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-kube-api-access-7jds9" (OuterVolumeSpecName: "kube-api-access-7jds9") pod "a6eb0a0d-5d9f-4c16-bec2-23421c47514b" (UID: "a6eb0a0d-5d9f-4c16-bec2-23421c47514b"). InnerVolumeSpecName "kube-api-access-7jds9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:39:16.176373 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.176316    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6eb0a0d-5d9f-4c16-bec2-23421c47514b" (UID: "a6eb0a0d-5d9f-4c16-bec2-23421c47514b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:39:16.275532 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.275495    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:16.275532 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.275526    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:16.275532 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.275534    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:16.275757 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.275551    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jds9\" (UniqueName: \"kubernetes.io/projected/a6eb0a0d-5d9f-4c16-bec2-23421c47514b-kube-api-access-7jds9\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:39:16.646921 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.646883    2571 generic.go:356] "Generic (PLEG): container finished" podID="a6eb0a0d-5d9f-4c16-bec2-23421c47514b" containerID="7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51" exitCode=0
Jul 30 11:39:16.647110 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.646926    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"
Jul 30 11:39:16.647110 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.646967    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq" event={"ID":"a6eb0a0d-5d9f-4c16-bec2-23421c47514b","Type":"ContainerDied","Data":"7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51"}
Jul 30 11:39:16.647110 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.647015    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq" event={"ID":"a6eb0a0d-5d9f-4c16-bec2-23421c47514b","Type":"ContainerDied","Data":"e70932face29861d4f1bfcdd2ce1758623ef481b4a38eca8f3ab788729ea466e"}
Jul 30 11:39:16.647110 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.647042    2571 scope.go:117] "RemoveContainer" containerID="7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51"
Jul 30 11:39:16.654998 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.654980    2571 scope.go:117] "RemoveContainer" containerID="7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51"
Jul 30 11:39:16.655286 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:39:16.655268    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51\": container with ID starting with 7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51 not found: ID does not exist" containerID="7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51"
Jul 30 11:39:16.655360 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.655294    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51"} err="failed to get container status \"7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51\": rpc error: code = NotFound desc = could not find container \"7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51\": container with ID starting with 7522367c052c5c0c7af245cf839cbfd5f6c789571c96e013799bfd6d80af8e51 not found: ID does not exist"
Jul 30 11:39:16.669940 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.669920    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"]
Jul 30 11:39:16.674114 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.674079    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/auth-test-model-945db9-kserve-66878fd66c-jllnq"]
Jul 30 11:39:16.854443 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.854409    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6901f37e-08bf-4acd-a1ea-213180056f26" path="/var/lib/kubelet/pods/6901f37e-08bf-4acd-a1ea-213180056f26/volumes"
Jul 30 11:39:16.854819 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:16.854806    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6eb0a0d-5d9f-4c16-bec2-23421c47514b" path="/var/lib/kubelet/pods/a6eb0a0d-5d9f-4c16-bec2-23421c47514b/volumes"
Jul 30 11:39:22.932766 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:22.932727    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:22.933166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:22.932780    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:22.945021 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:22.944991    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:23.688218 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:23.688183    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:39:35.088568 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.088532    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"]
Jul 30 11:39:35.089023 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.088830    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6901f37e-08bf-4acd-a1ea-213180056f26" containerName="main"
Jul 30 11:39:35.089023 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.088842    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6901f37e-08bf-4acd-a1ea-213180056f26" containerName="main"
Jul 30 11:39:35.089023 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.088859    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6eb0a0d-5d9f-4c16-bec2-23421c47514b" containerName="main"
Jul 30 11:39:35.089023 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.088868    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6eb0a0d-5d9f-4c16-bec2-23421c47514b" containerName="main"
Jul 30 11:39:35.089023 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.088918    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6eb0a0d-5d9f-4c16-bec2-23421c47514b" containerName="main"
Jul 30 11:39:35.089023 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.088926    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6901f37e-08bf-4acd-a1ea-213180056f26" containerName="main"
Jul 30 11:39:35.091732 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.091713    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.095930 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.095907    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"sub-test-model-945db9-kserve-self-signed-certs\""
Jul 30 11:39:35.096082 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.095962    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"default-dockercfg-l62n8\""
Jul 30 11:39:35.096082 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.095969    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"kube-root-ca.crt\""
Jul 30 11:39:35.096082 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.095970    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"openshift-service-ca.crt\""
Jul 30 11:39:35.104190 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.104166    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"]
Jul 30 11:39:35.109035 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.109012    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tmp-dir\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.109164 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.109061    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9pwh\" (UniqueName: \"kubernetes.io/projected/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-kube-api-access-r9pwh\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.109164 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.109138    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-model-cache\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.109164 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.109161    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-dshm\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.109292 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.109201    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-home\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.109292 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.109251    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tls-certs\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.210014 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.209985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tmp-dir\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.210192 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.210026    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9pwh\" (UniqueName: \"kubernetes.io/projected/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-kube-api-access-r9pwh\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.210192 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.210050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-model-cache\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.210192 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.210068    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-dshm\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.210192 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.210125    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-home\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.210445 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.210259    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tls-certs\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.210445 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.210380    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tmp-dir\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.210445 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.210418    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-model-cache\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.210705 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.210681    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-home\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.212318 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.212292    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-dshm\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.212611 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.212593    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tls-certs\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.219242 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.219225    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9pwh\" (UniqueName: \"kubernetes.io/projected/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-kube-api-access-r9pwh\") pod \"sub-test-model-945db9-kserve-77fb86c8b5-pt5mw\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") " pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.400648 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.400565    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:35.468021 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.467986    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:39:35.533411 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.533379    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"]
Jul 30 11:39:35.536071 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:39:35.536037    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb50ae584_2cc9_4d8e_839f_63ce1b26fce1.slice/crio-bfb0dff8dac61b3ad75f8c8a58e71a9ab866e681a33c67cd70da0ef283cc2b46 WatchSource:0}: Error finding container bfb0dff8dac61b3ad75f8c8a58e71a9ab866e681a33c67cd70da0ef283cc2b46: Status 404 returned error can't find the container with id bfb0dff8dac61b3ad75f8c8a58e71a9ab866e681a33c67cd70da0ef283cc2b46
Jul 30 11:39:35.727160 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.727124    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw" event={"ID":"b50ae584-2cc9-4d8e-839f-63ce1b26fce1","Type":"ContainerStarted","Data":"96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05"}
Jul 30 11:39:35.727326 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.727167    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw" event={"ID":"b50ae584-2cc9-4d8e-839f-63ce1b26fce1","Type":"ContainerStarted","Data":"bfb0dff8dac61b3ad75f8c8a58e71a9ab866e681a33c67cd70da0ef283cc2b46"}
Jul 30 11:39:35.745623 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:35.745578    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw" podStartSLOduration=0.745563126 podStartE2EDuration="745.563126ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:39:35.744575931 +0000 UTC m=+3081.465460191" watchObservedRunningTime="2026-07-30 11:39:35.745563126 +0000 UTC m=+3081.466447387"
Jul 30 11:39:37.686523 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:37.686491    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:39:45.400713 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:45.400680    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:45.401089 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:45.400726    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:45.412979 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:45.412962    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:39:45.778420 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:39:45.778390    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:40:13.014134 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:40:13.014072    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:40:15.024517 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:40:15.024482    2571 scope.go:117] "RemoveContainer" containerID="4e12ce0406eefcc1dc1bacae70ba8e9336cbe9c277692cf36c5a79a6c4f5c97b"
Jul 30 11:40:15.032013 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:40:15.031994    2571 scope.go:117] "RemoveContainer" containerID="d6e70747033d1385c01aaa21d439eda8dda44906e237d23dd401e7d8a8590baf"
Jul 30 11:40:24.190021 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:40:24.189986    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:40:52.084641 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:40:52.084608    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:40:56.871993 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:40:56.871960    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"]
Jul 30 11:40:56.893471 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:40:56.893440    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:40:57.864339 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:40:57.864308    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"]
Jul 30 11:40:57.877648 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:40:57.877620    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:41:02.484900 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:02.484860    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:41:11.890743 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:11.890695    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47" podUID="11906152-d87c-4245-ab2b-f784859ae101" containerName="main" containerID="cri-o://993f994ffa23824fdbe13bb1ab44027544bb4563da3f67f0ce25441f5752ca2e" gracePeriod=45
Jul 30 11:41:12.124341 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.124314    2571 generic.go:356] "Generic (PLEG): container finished" podID="11906152-d87c-4245-ab2b-f784859ae101" containerID="993f994ffa23824fdbe13bb1ab44027544bb4563da3f67f0ce25441f5752ca2e" exitCode=0
Jul 30 11:41:12.124483 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.124390    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47" event={"ID":"11906152-d87c-4245-ab2b-f784859ae101","Type":"ContainerDied","Data":"993f994ffa23824fdbe13bb1ab44027544bb4563da3f67f0ce25441f5752ca2e"}
Jul 30 11:41:12.124483 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.124443    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47" event={"ID":"11906152-d87c-4245-ab2b-f784859ae101","Type":"ContainerDied","Data":"0ff17431646a3038712fa1cb62ed7f1e0ecee115f85a936c66baa96b0c035917"}
Jul 30 11:41:12.124483 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.124459    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ff17431646a3038712fa1cb62ed7f1e0ecee115f85a936c66baa96b0c035917"
Jul 30 11:41:12.125713 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.125697    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:41:12.146035 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.145961    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-dshm\") pod \"11906152-d87c-4245-ab2b-f784859ae101\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") "
Jul 30 11:41:12.146035 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.146007    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-model-cache\") pod \"11906152-d87c-4245-ab2b-f784859ae101\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") "
Jul 30 11:41:12.146259 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.146039    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-home\") pod \"11906152-d87c-4245-ab2b-f784859ae101\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") "
Jul 30 11:41:12.146259 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.146072    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-tmp-dir\") pod \"11906152-d87c-4245-ab2b-f784859ae101\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") "
Jul 30 11:41:12.146259 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.146137    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11906152-d87c-4245-ab2b-f784859ae101-tls-certs\") pod \"11906152-d87c-4245-ab2b-f784859ae101\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") "
Jul 30 11:41:12.146259 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.146199    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t79z\" (UniqueName: \"kubernetes.io/projected/11906152-d87c-4245-ab2b-f784859ae101-kube-api-access-4t79z\") pod \"11906152-d87c-4245-ab2b-f784859ae101\" (UID: \"11906152-d87c-4245-ab2b-f784859ae101\") "
Jul 30 11:41:12.146745 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.146715    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-home" (OuterVolumeSpecName: "home") pod "11906152-d87c-4245-ab2b-f784859ae101" (UID: "11906152-d87c-4245-ab2b-f784859ae101"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:41:12.146886 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.146860    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11906152-d87c-4245-ab2b-f784859ae101" (UID: "11906152-d87c-4245-ab2b-f784859ae101"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:41:12.146957 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.146938    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-model-cache" (OuterVolumeSpecName: "model-cache") pod "11906152-d87c-4245-ab2b-f784859ae101" (UID: "11906152-d87c-4245-ab2b-f784859ae101"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:41:12.148378 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.148356    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-dshm" (OuterVolumeSpecName: "dshm") pod "11906152-d87c-4245-ab2b-f784859ae101" (UID: "11906152-d87c-4245-ab2b-f784859ae101"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:41:12.148889 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.148864    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11906152-d87c-4245-ab2b-f784859ae101-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11906152-d87c-4245-ab2b-f784859ae101" (UID: "11906152-d87c-4245-ab2b-f784859ae101"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:41:12.148975 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.148902    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11906152-d87c-4245-ab2b-f784859ae101-kube-api-access-4t79z" (OuterVolumeSpecName: "kube-api-access-4t79z") pod "11906152-d87c-4245-ab2b-f784859ae101" (UID: "11906152-d87c-4245-ab2b-f784859ae101"). InnerVolumeSpecName "kube-api-access-4t79z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:41:12.247435 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.247388    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t79z\" (UniqueName: \"kubernetes.io/projected/11906152-d87c-4245-ab2b-f784859ae101-kube-api-access-4t79z\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:12.247435 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.247426    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:12.247435 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.247439    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:12.247677 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.247453    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:12.247677 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.247465    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11906152-d87c-4245-ab2b-f784859ae101-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:12.247677 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.247476    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11906152-d87c-4245-ab2b-f784859ae101-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:12.882978 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:12.882912    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw" podUID="b50ae584-2cc9-4d8e-839f-63ce1b26fce1" containerName="main" containerID="cri-o://96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05" gracePeriod=45
Jul 30 11:41:13.125077 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.125017    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:41:13.129388 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.129361    2571 generic.go:356] "Generic (PLEG): container finished" podID="b50ae584-2cc9-4d8e-839f-63ce1b26fce1" containerID="96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05" exitCode=0
Jul 30 11:41:13.129499 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.129395    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"
Jul 30 11:41:13.129499 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.129448    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw" event={"ID":"b50ae584-2cc9-4d8e-839f-63ce1b26fce1","Type":"ContainerDied","Data":"96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05"}
Jul 30 11:41:13.129499 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.129489    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw" event={"ID":"b50ae584-2cc9-4d8e-839f-63ce1b26fce1","Type":"ContainerDied","Data":"bfb0dff8dac61b3ad75f8c8a58e71a9ab866e681a33c67cd70da0ef283cc2b46"}
Jul 30 11:41:13.129614 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.129499    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"
Jul 30 11:41:13.129614 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.129516    2571 scope.go:117] "RemoveContainer" containerID="96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05"
Jul 30 11:41:13.137240 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.137214    2571 scope.go:117] "RemoveContainer" containerID="96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05"
Jul 30 11:41:13.137495 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:41:13.137466    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05\": container with ID starting with 96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05 not found: ID does not exist" containerID="96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05"
Jul 30 11:41:13.137547 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.137502    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05"} err="failed to get container status \"96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05\": rpc error: code = NotFound desc = could not find container \"96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05\": container with ID starting with 96c690ad81e47e0695e79930149387e305cb989623d583fb54086222e2bede05 not found: ID does not exist"
Jul 30 11:41:13.154562 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.154539    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-model-cache\") pod \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") "
Jul 30 11:41:13.154716 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.154605    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tmp-dir\") pod \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") "
Jul 30 11:41:13.154716 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.154632    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-dshm\") pod \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") "
Jul 30 11:41:13.154716 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.154672    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tls-certs\") pod \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") "
Jul 30 11:41:13.154716 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.154699    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9pwh\" (UniqueName: \"kubernetes.io/projected/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-kube-api-access-r9pwh\") pod \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") "
Jul 30 11:41:13.154926 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.154725    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-home\") pod \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\" (UID: \"b50ae584-2cc9-4d8e-839f-63ce1b26fce1\") "
Jul 30 11:41:13.154926 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.154854    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b50ae584-2cc9-4d8e-839f-63ce1b26fce1" (UID: "b50ae584-2cc9-4d8e-839f-63ce1b26fce1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:41:13.154926 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.154872    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b50ae584-2cc9-4d8e-839f-63ce1b26fce1" (UID: "b50ae584-2cc9-4d8e-839f-63ce1b26fce1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:41:13.155082 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.154995    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:13.155082 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.155039    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:13.155225 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.155076    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-home" (OuterVolumeSpecName: "home") pod "b50ae584-2cc9-4d8e-839f-63ce1b26fce1" (UID: "b50ae584-2cc9-4d8e-839f-63ce1b26fce1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:41:13.156571 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.156542    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"]
Jul 30 11:41:13.157128 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.157081    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-kube-api-access-r9pwh" (OuterVolumeSpecName: "kube-api-access-r9pwh") pod "b50ae584-2cc9-4d8e-839f-63ce1b26fce1" (UID: "b50ae584-2cc9-4d8e-839f-63ce1b26fce1"). InnerVolumeSpecName "kube-api-access-r9pwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:41:13.157248 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.157158    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-dshm" (OuterVolumeSpecName: "dshm") pod "b50ae584-2cc9-4d8e-839f-63ce1b26fce1" (UID: "b50ae584-2cc9-4d8e-839f-63ce1b26fce1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:41:13.157289 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.157265    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b50ae584-2cc9-4d8e-839f-63ce1b26fce1" (UID: "b50ae584-2cc9-4d8e-839f-63ce1b26fce1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:41:13.161802 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.161701    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/sub-test-model-060e9f-kserve-6f45ddc4dc-bpg47"]
Jul 30 11:41:13.256223 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.256190    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:13.256223 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.256217    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:13.256223 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.256228    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9pwh\" (UniqueName: \"kubernetes.io/projected/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-kube-api-access-r9pwh\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:13.256455 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.256237    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b50ae584-2cc9-4d8e-839f-63ce1b26fce1-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:41:13.459842 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.459816    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"]
Jul 30 11:41:13.475883 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:13.475856    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/sub-test-model-945db9-kserve-77fb86c8b5-pt5mw"]
Jul 30 11:41:14.783918 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.783877    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"]
Jul 30 11:41:14.784388 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.784286    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50ae584-2cc9-4d8e-839f-63ce1b26fce1" containerName="main"
Jul 30 11:41:14.784388 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.784305    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50ae584-2cc9-4d8e-839f-63ce1b26fce1" containerName="main"
Jul 30 11:41:14.784388 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.784326    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11906152-d87c-4245-ab2b-f784859ae101" containerName="main"
Jul 30 11:41:14.784388 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.784335    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="11906152-d87c-4245-ab2b-f784859ae101" containerName="main"
Jul 30 11:41:14.784590 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.784409    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50ae584-2cc9-4d8e-839f-63ce1b26fce1" containerName="main"
Jul 30 11:41:14.784590 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.784420    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="11906152-d87c-4245-ab2b-f784859ae101" containerName="main"
Jul 30 11:41:14.788337 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.788315    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.791292 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.791271    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-060e9f\"/\"openshift-service-ca.crt\""
Jul 30 11:41:14.791292 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.791289    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-060e9f\"/\"kube-root-ca.crt\""
Jul 30 11:41:14.792432 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.792411    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-060e9f\"/\"default-dockercfg-xm2jv\""
Jul 30 11:41:14.792535 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.792437    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-060e9f\"/\"rate-test-model-060e9f-kserve-self-signed-certs\""
Jul 30 11:41:14.796579 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.796547    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"]
Jul 30 11:41:14.812640 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.812608    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:41:14.855665 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.855632    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11906152-d87c-4245-ab2b-f784859ae101" path="/var/lib/kubelet/pods/11906152-d87c-4245-ab2b-f784859ae101/volumes"
Jul 30 11:41:14.856112 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.856081    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50ae584-2cc9-4d8e-839f-63ce1b26fce1" path="/var/lib/kubelet/pods/b50ae584-2cc9-4d8e-839f-63ce1b26fce1/volumes"
Jul 30 11:41:14.870999 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.870972    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-dshm\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.871124 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.871002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-model-cache\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.871124 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.871020    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tls-certs\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.871124 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.871047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8knv7\" (UniqueName: \"kubernetes.io/projected/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-kube-api-access-8knv7\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.871245 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.871140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tmp-dir\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.871245 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.871163    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-home\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.972225 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.972192    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tmp-dir\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.972393 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.972239    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-home\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.972393 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.972291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-dshm\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.972393 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.972320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-model-cache\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.972393 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.972344    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tls-certs\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.972579 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.972374    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8knv7\" (UniqueName: \"kubernetes.io/projected/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-kube-api-access-8knv7\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.972693 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.972662    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tmp-dir\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.972829 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.972809    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-model-cache\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.972893 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.972841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-home\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.974607 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.974585    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-dshm\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.974880 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.974861    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tls-certs\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:14.981137 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:14.981087    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8knv7\" (UniqueName: \"kubernetes.io/projected/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-kube-api-access-8knv7\") pod \"rate-test-model-060e9f-kserve-57657549f5-px8bj\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") " pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:15.098827 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:15.098797    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:15.226198 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:15.226170    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"]
Jul 30 11:41:15.228482 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:41:15.228453    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4b0f1ee_117d_4fd7_b65c_1e745007a8e9.slice/crio-85b8f20910c82f8cacd62cc2591bbd0191dfa014718657c80112f3a8c7e0f486 WatchSource:0}: Error finding container 85b8f20910c82f8cacd62cc2591bbd0191dfa014718657c80112f3a8c7e0f486: Status 404 returned error can't find the container with id 85b8f20910c82f8cacd62cc2591bbd0191dfa014718657c80112f3a8c7e0f486
Jul 30 11:41:16.143752 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:16.143717    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj" event={"ID":"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9","Type":"ContainerStarted","Data":"ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08"}
Jul 30 11:41:16.144173 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:16.143767    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj" event={"ID":"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9","Type":"ContainerStarted","Data":"85b8f20910c82f8cacd62cc2591bbd0191dfa014718657c80112f3a8c7e0f486"}
Jul 30 11:41:16.161729 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:16.161681    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj" podStartSLOduration=2.161666948 podStartE2EDuration="2.161666948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:41:16.160816112 +0000 UTC m=+3181.881700373" watchObservedRunningTime="2026-07-30 11:41:16.161666948 +0000 UTC m=+3181.882551209"
Jul 30 11:41:17.205756 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:17.205723    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:41:25.099203 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:25.099168    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:25.099686 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:25.099222    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:25.111512 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:25.111481    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:25.189854 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:25.189826    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:41:29.406200 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:29.406167    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:41:34.554408 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:34.554361    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:41:41.169729 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.169696    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"]
Jul 30 11:41:41.172650 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.172627    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.175488 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.175465    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"openshift-service-ca.crt\""
Jul 30 11:41:41.176872 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.176842    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"default-dockercfg-l62n8\""
Jul 30 11:41:41.176872 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.176866    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"kube-root-ca.crt\""
Jul 30 11:41:41.177038 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.176867    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-945db9\"/\"rate-test-model-945db9-kserve-self-signed-certs\""
Jul 30 11:41:41.181996 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.181958    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"]
Jul 30 11:41:41.203873 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.203851    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:41:41.287288 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.287260    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-home\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.287288 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.287290    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-model-cache\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.287482 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.287307    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tmp-dir\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.287482 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.287330    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-dshm\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.287482 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.287426    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2hq\" (UniqueName: \"kubernetes.io/projected/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-kube-api-access-6k2hq\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.287603 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.287487    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tls-certs\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.387860 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.387830    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-home\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.388039 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.387867    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-model-cache\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.388039 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.387892    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tmp-dir\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.388039 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.387948    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-dshm\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.388039 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.387997    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2hq\" (UniqueName: \"kubernetes.io/projected/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-kube-api-access-6k2hq\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.388296 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.388045    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tls-certs\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.388296 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.388287    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-home\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.388407 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.388341    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tmp-dir\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.388407 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.388376    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-model-cache\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.390245 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.390222    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-dshm\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.390592 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.390575    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tls-certs\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.397263 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.397239    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2hq\" (UniqueName: \"kubernetes.io/projected/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-kube-api-access-6k2hq\") pod \"rate-test-model-945db9-kserve-6c9fffb6c-fbdcz\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") " pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.482618 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.482587    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:41.609528 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:41.609495    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"]
Jul 30 11:41:41.612646 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:41:41.612610    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee09f3fd_24c0_4f9b_a970_b41e33ab12c5.slice/crio-1963f2d341009b9d53636d28db469025d0bea373ff8cd72e6d68ecbc2d62d3de WatchSource:0}: Error finding container 1963f2d341009b9d53636d28db469025d0bea373ff8cd72e6d68ecbc2d62d3de: Status 404 returned error can't find the container with id 1963f2d341009b9d53636d28db469025d0bea373ff8cd72e6d68ecbc2d62d3de
Jul 30 11:41:42.247050 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:42.247012    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz" event={"ID":"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5","Type":"ContainerStarted","Data":"fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70"}
Jul 30 11:41:42.247487 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:42.247061    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz" event={"ID":"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5","Type":"ContainerStarted","Data":"1963f2d341009b9d53636d28db469025d0bea373ff8cd72e6d68ecbc2d62d3de"}
Jul 30 11:41:42.266817 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:42.266753    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz" podStartSLOduration=1.266739016 podStartE2EDuration="1.266739016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:41:42.265772504 +0000 UTC m=+3207.986656770" watchObservedRunningTime="2026-07-30 11:41:42.266739016 +0000 UTC m=+3207.987623276"
Jul 30 11:41:43.684016 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:43.683980    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:41:51.483112 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:51.483065    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:51.483531 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:51.483163    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:51.495759 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:51.495731    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:41:52.300856 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:41:52.300826    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:42:21.093393 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:21.093354    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:42:29.394894 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:29.394853    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:42:46.191220 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:46.191185    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:42:51.263205 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:51.263168    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"]
Jul 30 11:42:51.284237 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:51.284205    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:42:52.091400 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:52.091368    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"]
Jul 30 11:42:52.109551 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:52.109499    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:42:53.133355 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.133320    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7382-139810755914800"]
Jul 30 11:42:53.136427 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.136411    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755914800"
Jul 30 11:42:53.139248 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.139224    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dvncz\""
Jul 30 11:42:53.143240 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.142973    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810755914800"]
Jul 30 11:42:53.179362 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.179335    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qq8r\" (UniqueName: \"kubernetes.io/projected/19e18119-4e60-4d49-8bab-a4c5d1958e30-kube-api-access-5qq8r\") pod \"test-curl-7382-139810755914800\" (UID: \"19e18119-4e60-4d49-8bab-a4c5d1958e30\") " pod="opendatahub/test-curl-7382-139810755914800"
Jul 30 11:42:53.279883 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.279856    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq8r\" (UniqueName: \"kubernetes.io/projected/19e18119-4e60-4d49-8bab-a4c5d1958e30-kube-api-access-5qq8r\") pod \"test-curl-7382-139810755914800\" (UID: \"19e18119-4e60-4d49-8bab-a4c5d1958e30\") " pod="opendatahub/test-curl-7382-139810755914800"
Jul 30 11:42:53.291502 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.291475    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qq8r\" (UniqueName: \"kubernetes.io/projected/19e18119-4e60-4d49-8bab-a4c5d1958e30-kube-api-access-5qq8r\") pod \"test-curl-7382-139810755914800\" (UID: \"19e18119-4e60-4d49-8bab-a4c5d1958e30\") " pod="opendatahub/test-curl-7382-139810755914800"
Jul 30 11:42:53.446250 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.446217    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755914800"
Jul 30 11:42:53.576398 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.576365    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810755914800"]
Jul 30 11:42:53.579945 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:42:53.579901    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19e18119_4e60_4d49_8bab_a4c5d1958e30.slice/crio-f94079c9523e50f9dd7f7ce4fadcbd7279db2daa179439c09226c858e2d6e267 WatchSource:0}: Error finding container f94079c9523e50f9dd7f7ce4fadcbd7279db2daa179439c09226c858e2d6e267: Status 404 returned error can't find the container with id f94079c9523e50f9dd7f7ce4fadcbd7279db2daa179439c09226c858e2d6e267
Jul 30 11:42:53.581619 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.581603    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 11:42:53.899469 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:53.899383    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:42:54.547880 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:54.547789    2571 generic.go:356] "Generic (PLEG): container finished" podID="19e18119-4e60-4d49-8bab-a4c5d1958e30" containerID="84ce42d70be0a8509f0074bbb9be30a6206210aaac1e6ba5000d5e9eca1bd62c" exitCode=0
Jul 30 11:42:54.548268 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:54.547878    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810755914800" event={"ID":"19e18119-4e60-4d49-8bab-a4c5d1958e30","Type":"ContainerDied","Data":"84ce42d70be0a8509f0074bbb9be30a6206210aaac1e6ba5000d5e9eca1bd62c"}
Jul 30 11:42:54.548268 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:54.547937    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810755914800" event={"ID":"19e18119-4e60-4d49-8bab-a4c5d1958e30","Type":"ContainerStarted","Data":"f94079c9523e50f9dd7f7ce4fadcbd7279db2daa179439c09226c858e2d6e267"}
Jul 30 11:42:55.681023 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.680997    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755914800"
Jul 30 11:42:55.699782 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.699756    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qq8r\" (UniqueName: \"kubernetes.io/projected/19e18119-4e60-4d49-8bab-a4c5d1958e30-kube-api-access-5qq8r\") pod \"19e18119-4e60-4d49-8bab-a4c5d1958e30\" (UID: \"19e18119-4e60-4d49-8bab-a4c5d1958e30\") "
Jul 30 11:42:55.701922 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.701893    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e18119-4e60-4d49-8bab-a4c5d1958e30-kube-api-access-5qq8r" (OuterVolumeSpecName: "kube-api-access-5qq8r") pod "19e18119-4e60-4d49-8bab-a4c5d1958e30" (UID: "19e18119-4e60-4d49-8bab-a4c5d1958e30"). InnerVolumeSpecName "kube-api-access-5qq8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:42:55.703529 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.703502    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7382-139810755914800_19e18119-4e60-4d49-8bab-a4c5d1958e30/test-curl-7382-139810755914800/0.log"
Jul 30 11:42:55.727944 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.727917    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7382-139810755914800"]
Jul 30 11:42:55.731767 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.731747    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7382-139810755914800"]
Jul 30 11:42:55.800709 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.800683    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qq8r\" (UniqueName: \"kubernetes.io/projected/19e18119-4e60-4d49-8bab-a4c5d1958e30-kube-api-access-5qq8r\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:42:55.848283 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.848219    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7382-139810755913392"]
Jul 30 11:42:55.848524 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.848511    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e18119-4e60-4d49-8bab-a4c5d1958e30" containerName="test-curl-7382-139810755914800"
Jul 30 11:42:55.848568 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.848526    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e18119-4e60-4d49-8bab-a4c5d1958e30" containerName="test-curl-7382-139810755914800"
Jul 30 11:42:55.848603 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.848586    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e18119-4e60-4d49-8bab-a4c5d1958e30" containerName="test-curl-7382-139810755914800"
Jul 30 11:42:55.851545 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.851524    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755913392"
Jul 30 11:42:55.858683 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.858444    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810755913392"]
Jul 30 11:42:55.901772 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:55.901744    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9ndq\" (UniqueName: \"kubernetes.io/projected/da9d9481-ce3f-4332-8db3-be75008031b9-kube-api-access-m9ndq\") pod \"test-curl-7382-139810755913392\" (UID: \"da9d9481-ce3f-4332-8db3-be75008031b9\") " pod="opendatahub/test-curl-7382-139810755913392"
Jul 30 11:42:56.002967 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:56.002935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9ndq\" (UniqueName: \"kubernetes.io/projected/da9d9481-ce3f-4332-8db3-be75008031b9-kube-api-access-m9ndq\") pod \"test-curl-7382-139810755913392\" (UID: \"da9d9481-ce3f-4332-8db3-be75008031b9\") " pod="opendatahub/test-curl-7382-139810755913392"
Jul 30 11:42:56.011716 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:56.011687    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9ndq\" (UniqueName: \"kubernetes.io/projected/da9d9481-ce3f-4332-8db3-be75008031b9-kube-api-access-m9ndq\") pod \"test-curl-7382-139810755913392\" (UID: \"da9d9481-ce3f-4332-8db3-be75008031b9\") " pod="opendatahub/test-curl-7382-139810755913392"
Jul 30 11:42:56.161084 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:56.160997    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755913392"
Jul 30 11:42:56.494330 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:56.494299    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810755913392"]
Jul 30 11:42:56.496597 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:42:56.496566    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda9d9481_ce3f_4332_8db3_be75008031b9.slice/crio-702150bd391f1b9553a3732bf54940883240cd7199aac0c55da40aae0734ebd8 WatchSource:0}: Error finding container 702150bd391f1b9553a3732bf54940883240cd7199aac0c55da40aae0734ebd8: Status 404 returned error can't find the container with id 702150bd391f1b9553a3732bf54940883240cd7199aac0c55da40aae0734ebd8
Jul 30 11:42:56.559142 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:56.559113    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810755913392" event={"ID":"da9d9481-ce3f-4332-8db3-be75008031b9","Type":"ContainerStarted","Data":"702150bd391f1b9553a3732bf54940883240cd7199aac0c55da40aae0734ebd8"}
Jul 30 11:42:56.560413 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:56.560392    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755914800"
Jul 30 11:42:56.560533 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:56.560417    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f94079c9523e50f9dd7f7ce4fadcbd7279db2daa179439c09226c858e2d6e267"
Jul 30 11:42:56.571501 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:42:56.571465    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7382-139810755914800\" is forbidden: User \"system:node:ip-10-0-128-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-243.ec2.internal' and this object" podUID="19e18119-4e60-4d49-8bab-a4c5d1958e30" pod="opendatahub/test-curl-7382-139810755914800"
Jul 30 11:42:56.855489 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:56.855460    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19e18119-4e60-4d49-8bab-a4c5d1958e30" path="/var/lib/kubelet/pods/19e18119-4e60-4d49-8bab-a4c5d1958e30/volumes"
Jul 30 11:42:57.565911 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:57.565879    2571 generic.go:356] "Generic (PLEG): container finished" podID="da9d9481-ce3f-4332-8db3-be75008031b9" containerID="41b4aec1e41619c526b6da49c38f8080a3234423592702ebccb372549a0789dd" exitCode=0
Jul 30 11:42:57.566066 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:57.565975    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810755913392" event={"ID":"da9d9481-ce3f-4332-8db3-be75008031b9","Type":"ContainerDied","Data":"41b4aec1e41619c526b6da49c38f8080a3234423592702ebccb372549a0789dd"}
Jul 30 11:42:58.699491 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.699468    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755913392"
Jul 30 11:42:58.719368 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.719339    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7382-139810755913392_da9d9481-ce3f-4332-8db3-be75008031b9/test-curl-7382-139810755913392/0.log"
Jul 30 11:42:58.725446 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.725425    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9ndq\" (UniqueName: \"kubernetes.io/projected/da9d9481-ce3f-4332-8db3-be75008031b9-kube-api-access-m9ndq\") pod \"da9d9481-ce3f-4332-8db3-be75008031b9\" (UID: \"da9d9481-ce3f-4332-8db3-be75008031b9\") "
Jul 30 11:42:58.727471 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.727443    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9d9481-ce3f-4332-8db3-be75008031b9-kube-api-access-m9ndq" (OuterVolumeSpecName: "kube-api-access-m9ndq") pod "da9d9481-ce3f-4332-8db3-be75008031b9" (UID: "da9d9481-ce3f-4332-8db3-be75008031b9"). InnerVolumeSpecName "kube-api-access-m9ndq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:42:58.740377 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.740355    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7382-139810755913392"]
Jul 30 11:42:58.743846 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.743823    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7382-139810755913392"]
Jul 30 11:42:58.826291 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.826231    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9ndq\" (UniqueName: \"kubernetes.io/projected/da9d9481-ce3f-4332-8db3-be75008031b9-kube-api-access-m9ndq\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:42:58.854371 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.854343    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da9d9481-ce3f-4332-8db3-be75008031b9" path="/var/lib/kubelet/pods/da9d9481-ce3f-4332-8db3-be75008031b9/volumes"
Jul 30 11:42:58.912664 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.912637    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7382-139810756021392"]
Jul 30 11:42:58.912939 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.912927    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9d9481-ce3f-4332-8db3-be75008031b9" containerName="test-curl-7382-139810755913392"
Jul 30 11:42:58.912984 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.912942    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9d9481-ce3f-4332-8db3-be75008031b9" containerName="test-curl-7382-139810755913392"
Jul 30 11:42:58.913017 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.912986    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9d9481-ce3f-4332-8db3-be75008031b9" containerName="test-curl-7382-139810755913392"
Jul 30 11:42:58.915780 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.915765    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:42:58.923170 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.923148    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810756021392"]
Jul 30 11:42:58.926982 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:58.926956    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6kq\" (UniqueName: \"kubernetes.io/projected/d4df0f08-a658-4f25-9229-9191aa8e1c6f-kube-api-access-qf6kq\") pod \"test-curl-7382-139810756021392\" (UID: \"d4df0f08-a658-4f25-9229-9191aa8e1c6f\") " pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:42:59.027928 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:59.027891    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6kq\" (UniqueName: \"kubernetes.io/projected/d4df0f08-a658-4f25-9229-9191aa8e1c6f-kube-api-access-qf6kq\") pod \"test-curl-7382-139810756021392\" (UID: \"d4df0f08-a658-4f25-9229-9191aa8e1c6f\") " pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:42:59.036864 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:59.036841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6kq\" (UniqueName: \"kubernetes.io/projected/d4df0f08-a658-4f25-9229-9191aa8e1c6f-kube-api-access-qf6kq\") pod \"test-curl-7382-139810756021392\" (UID: \"d4df0f08-a658-4f25-9229-9191aa8e1c6f\") " pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:42:59.224912 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:59.224870    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:42:59.349501 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:59.349475    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810756021392"]
Jul 30 11:42:59.351357 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:42:59.351332    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4df0f08_a658_4f25_9229_9191aa8e1c6f.slice/crio-f7a256cd531294f8726b6995a4bde61adbfd9eeb79cb0c05ef203f5337f5acef WatchSource:0}: Error finding container f7a256cd531294f8726b6995a4bde61adbfd9eeb79cb0c05ef203f5337f5acef: Status 404 returned error can't find the container with id f7a256cd531294f8726b6995a4bde61adbfd9eeb79cb0c05ef203f5337f5acef
Jul 30 11:42:59.575203 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:59.575117    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810756021392" event={"ID":"d4df0f08-a658-4f25-9229-9191aa8e1c6f","Type":"ContainerStarted","Data":"f7a256cd531294f8726b6995a4bde61adbfd9eeb79cb0c05ef203f5337f5acef"}
Jul 30 11:42:59.576320 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:59.576298    2571 scope.go:117] "RemoveContainer" containerID="41b4aec1e41619c526b6da49c38f8080a3234423592702ebccb372549a0789dd"
Jul 30 11:42:59.576457 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:42:59.576321    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755913392"
Jul 30 11:43:00.581266 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:00.581227    2571 generic.go:356] "Generic (PLEG): container finished" podID="d4df0f08-a658-4f25-9229-9191aa8e1c6f" containerID="5f6a17d86920e4991a3e6314061861ec156d4cd54eb59516036259786bab193f" exitCode=0
Jul 30 11:43:00.581661 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:00.581319    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810756021392" event={"ID":"d4df0f08-a658-4f25-9229-9191aa8e1c6f","Type":"ContainerDied","Data":"5f6a17d86920e4991a3e6314061861ec156d4cd54eb59516036259786bab193f"}
Jul 30 11:43:01.714604 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.714581    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:43:01.735026 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.735005    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7382-139810756021392_d4df0f08-a658-4f25-9229-9191aa8e1c6f/test-curl-7382-139810756021392/0.log"
Jul 30 11:43:01.749038 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.749018    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf6kq\" (UniqueName: \"kubernetes.io/projected/d4df0f08-a658-4f25-9229-9191aa8e1c6f-kube-api-access-qf6kq\") pod \"d4df0f08-a658-4f25-9229-9191aa8e1c6f\" (UID: \"d4df0f08-a658-4f25-9229-9191aa8e1c6f\") "
Jul 30 11:43:01.751043 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.751014    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4df0f08-a658-4f25-9229-9191aa8e1c6f-kube-api-access-qf6kq" (OuterVolumeSpecName: "kube-api-access-qf6kq") pod "d4df0f08-a658-4f25-9229-9191aa8e1c6f" (UID: "d4df0f08-a658-4f25-9229-9191aa8e1c6f"). InnerVolumeSpecName "kube-api-access-qf6kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:43:01.757112 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.757078    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7382-139810756021392"]
Jul 30 11:43:01.762757 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.762736    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7382-139810756021392"]
Jul 30 11:43:01.850505 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.850443    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf6kq\" (UniqueName: \"kubernetes.io/projected/d4df0f08-a658-4f25-9229-9191aa8e1c6f-kube-api-access-qf6kq\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:01.885912 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.885880    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7382-139810756021392"]
Jul 30 11:43:01.886237 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.886220    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4df0f08-a658-4f25-9229-9191aa8e1c6f" containerName="test-curl-7382-139810756021392"
Jul 30 11:43:01.886322 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.886239    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4df0f08-a658-4f25-9229-9191aa8e1c6f" containerName="test-curl-7382-139810756021392"
Jul 30 11:43:01.886378 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.886346    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4df0f08-a658-4f25-9229-9191aa8e1c6f" containerName="test-curl-7382-139810756021392"
Jul 30 11:43:01.889088 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.889068    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:43:01.897541 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.897518    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810756021392"]
Jul 30 11:43:01.951448 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:01.951421    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxhdh\" (UniqueName: \"kubernetes.io/projected/24d377e1-4f1b-4e37-bab1-a95352137b46-kube-api-access-sxhdh\") pod \"test-curl-7382-139810756021392\" (UID: \"24d377e1-4f1b-4e37-bab1-a95352137b46\") " pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:43:02.052063 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:02.052024    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxhdh\" (UniqueName: \"kubernetes.io/projected/24d377e1-4f1b-4e37-bab1-a95352137b46-kube-api-access-sxhdh\") pod \"test-curl-7382-139810756021392\" (UID: \"24d377e1-4f1b-4e37-bab1-a95352137b46\") " pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:43:02.061210 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:02.061191    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxhdh\" (UniqueName: \"kubernetes.io/projected/24d377e1-4f1b-4e37-bab1-a95352137b46-kube-api-access-sxhdh\") pod \"test-curl-7382-139810756021392\" (UID: \"24d377e1-4f1b-4e37-bab1-a95352137b46\") " pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:43:02.199274 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:02.199249    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:43:02.323076 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:02.323051    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810756021392"]
Jul 30 11:43:02.325491 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:43:02.325462    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24d377e1_4f1b_4e37_bab1_a95352137b46.slice/crio-354f43aca2952c0ea5d9ae3bbefff1d55c251df17337413c551d4ef4e2ccf679 WatchSource:0}: Error finding container 354f43aca2952c0ea5d9ae3bbefff1d55c251df17337413c551d4ef4e2ccf679: Status 404 returned error can't find the container with id 354f43aca2952c0ea5d9ae3bbefff1d55c251df17337413c551d4ef4e2ccf679
Jul 30 11:43:02.591561 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:02.591479    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:43:02.591561 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:02.591499    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7a256cd531294f8726b6995a4bde61adbfd9eeb79cb0c05ef203f5337f5acef"
Jul 30 11:43:02.592656 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:02.592628    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810756021392" event={"ID":"24d377e1-4f1b-4e37-bab1-a95352137b46","Type":"ContainerStarted","Data":"354f43aca2952c0ea5d9ae3bbefff1d55c251df17337413c551d4ef4e2ccf679"}
Jul 30 11:43:02.604285 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:02.604251    2571 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7382-139810756021392" oldPodUID="d4df0f08-a658-4f25-9229-9191aa8e1c6f" podUID="24d377e1-4f1b-4e37-bab1-a95352137b46"
Jul 30 11:43:02.855057 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:02.854983    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4df0f08-a658-4f25-9229-9191aa8e1c6f" path="/var/lib/kubelet/pods/d4df0f08-a658-4f25-9229-9191aa8e1c6f/volumes"
Jul 30 11:43:03.597766 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:03.597732    2571 generic.go:356] "Generic (PLEG): container finished" podID="24d377e1-4f1b-4e37-bab1-a95352137b46" containerID="bca2a4bf179c9251436344bb6da82fd8adb752f5182aa29341f838adb934a187" exitCode=0
Jul 30 11:43:03.597933 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:03.597810    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810756021392" event={"ID":"24d377e1-4f1b-4e37-bab1-a95352137b46","Type":"ContainerDied","Data":"bca2a4bf179c9251436344bb6da82fd8adb752f5182aa29341f838adb934a187"}
Jul 30 11:43:04.725535 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.725507    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:43:04.745844 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.745813    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7382-139810756021392_24d377e1-4f1b-4e37-bab1-a95352137b46/test-curl-7382-139810756021392/0.log"
Jul 30 11:43:04.767563 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.767532    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7382-139810756021392"]
Jul 30 11:43:04.771429 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.771388    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxhdh\" (UniqueName: \"kubernetes.io/projected/24d377e1-4f1b-4e37-bab1-a95352137b46-kube-api-access-sxhdh\") pod \"24d377e1-4f1b-4e37-bab1-a95352137b46\" (UID: \"24d377e1-4f1b-4e37-bab1-a95352137b46\") "
Jul 30 11:43:04.773448 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.773417    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d377e1-4f1b-4e37-bab1-a95352137b46-kube-api-access-sxhdh" (OuterVolumeSpecName: "kube-api-access-sxhdh") pod "24d377e1-4f1b-4e37-bab1-a95352137b46" (UID: "24d377e1-4f1b-4e37-bab1-a95352137b46"). InnerVolumeSpecName "kube-api-access-sxhdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:43:04.773532 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.773475    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7382-139810756021392"]
Jul 30 11:43:04.854899 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.854838    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d377e1-4f1b-4e37-bab1-a95352137b46" path="/var/lib/kubelet/pods/24d377e1-4f1b-4e37-bab1-a95352137b46/volumes"
Jul 30 11:43:04.872825 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.872802    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxhdh\" (UniqueName: \"kubernetes.io/projected/24d377e1-4f1b-4e37-bab1-a95352137b46-kube-api-access-sxhdh\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:04.890414 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.890390    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7382-139810755936304"]
Jul 30 11:43:04.890654 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.890641    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d377e1-4f1b-4e37-bab1-a95352137b46" containerName="test-curl-7382-139810756021392"
Jul 30 11:43:04.890698 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.890656    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d377e1-4f1b-4e37-bab1-a95352137b46" containerName="test-curl-7382-139810756021392"
Jul 30 11:43:04.890736 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.890713    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d377e1-4f1b-4e37-bab1-a95352137b46" containerName="test-curl-7382-139810756021392"
Jul 30 11:43:04.893516 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.893502    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:04.900339 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.900315    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810755936304"]
Jul 30 11:43:04.974046 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:04.974015    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dqg5\" (UniqueName: \"kubernetes.io/projected/667989e1-6f19-4ce3-9371-7697df061872-kube-api-access-8dqg5\") pod \"test-curl-7382-139810755936304\" (UID: \"667989e1-6f19-4ce3-9371-7697df061872\") " pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:05.074770 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:05.074732    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqg5\" (UniqueName: \"kubernetes.io/projected/667989e1-6f19-4ce3-9371-7697df061872-kube-api-access-8dqg5\") pod \"test-curl-7382-139810755936304\" (UID: \"667989e1-6f19-4ce3-9371-7697df061872\") " pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:05.083479 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:05.083448    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqg5\" (UniqueName: \"kubernetes.io/projected/667989e1-6f19-4ce3-9371-7697df061872-kube-api-access-8dqg5\") pod \"test-curl-7382-139810755936304\" (UID: \"667989e1-6f19-4ce3-9371-7697df061872\") " pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:05.202345 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:05.202315    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:05.330000 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:05.329975    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810755936304"]
Jul 30 11:43:05.332320 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:43:05.332292    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod667989e1_6f19_4ce3_9371_7697df061872.slice/crio-09e5af9cbbfff0af2aa162298f010f80b5cbbe8d3d5931aee0827f44dd7f518f WatchSource:0}: Error finding container 09e5af9cbbfff0af2aa162298f010f80b5cbbe8d3d5931aee0827f44dd7f518f: Status 404 returned error can't find the container with id 09e5af9cbbfff0af2aa162298f010f80b5cbbe8d3d5931aee0827f44dd7f518f
Jul 30 11:43:05.607011 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:05.606984    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810755936304" event={"ID":"667989e1-6f19-4ce3-9371-7697df061872","Type":"ContainerStarted","Data":"09e5af9cbbfff0af2aa162298f010f80b5cbbe8d3d5931aee0827f44dd7f518f"}
Jul 30 11:43:05.608258 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:05.608239    2571 scope.go:117] "RemoveContainer" containerID="bca2a4bf179c9251436344bb6da82fd8adb752f5182aa29341f838adb934a187"
Jul 30 11:43:05.608351 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:05.608271    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810756021392"
Jul 30 11:43:06.281635 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.281596    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj" podUID="e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" containerName="main" containerID="cri-o://ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08" gracePeriod=45
Jul 30 11:43:06.531311 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.531282    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:43:06.587918 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.587848    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-home\") pod \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") "
Jul 30 11:43:06.587918 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.587883    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-dshm\") pod \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") "
Jul 30 11:43:06.587918 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.587901    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tmp-dir\") pod \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") "
Jul 30 11:43:06.588181 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.587950    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-model-cache\") pod \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") "
Jul 30 11:43:06.588181 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.587997    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tls-certs\") pod \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") "
Jul 30 11:43:06.588181 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.588026    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8knv7\" (UniqueName: \"kubernetes.io/projected/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-kube-api-access-8knv7\") pod \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\" (UID: \"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9\") "
Jul 30 11:43:06.588181 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.588136    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-home" (OuterVolumeSpecName: "home") pod "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" (UID: "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:43:06.588369 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.588241    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" (UID: "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:43:06.588369 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.588278    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" (UID: "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:43:06.588369 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.588301    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:06.588369 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.588314    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:06.589954 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.589935    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-dshm" (OuterVolumeSpecName: "dshm") pod "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" (UID: "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:43:06.590053 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.590027    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" (UID: "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:43:06.590143 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.590119    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-kube-api-access-8knv7" (OuterVolumeSpecName: "kube-api-access-8knv7") pod "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" (UID: "e4b0f1ee-117d-4fd7-b65c-1e745007a8e9"). InnerVolumeSpecName "kube-api-access-8knv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:43:06.614072 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.614046    2571 generic.go:356] "Generic (PLEG): container finished" podID="e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" containerID="ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08" exitCode=0
Jul 30 11:43:06.614205 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.614115    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"
Jul 30 11:43:06.614205 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.614138    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj" event={"ID":"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9","Type":"ContainerDied","Data":"ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08"}
Jul 30 11:43:06.614205 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.614188    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj" event={"ID":"e4b0f1ee-117d-4fd7-b65c-1e745007a8e9","Type":"ContainerDied","Data":"85b8f20910c82f8cacd62cc2591bbd0191dfa014718657c80112f3a8c7e0f486"}
Jul 30 11:43:06.614347 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.614214    2571 scope.go:117] "RemoveContainer" containerID="ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08"
Jul 30 11:43:06.616619 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.616596    2571 generic.go:356] "Generic (PLEG): container finished" podID="667989e1-6f19-4ce3-9371-7697df061872" containerID="ca745a7ed2539f66f72b5f3db5950d6c31c40287ec357d67c81d303129e66ecf" exitCode=0
Jul 30 11:43:06.616716 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.616648    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810755936304" event={"ID":"667989e1-6f19-4ce3-9371-7697df061872","Type":"ContainerDied","Data":"ca745a7ed2539f66f72b5f3db5950d6c31c40287ec357d67c81d303129e66ecf"}
Jul 30 11:43:06.622481 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.622459    2571 scope.go:117] "RemoveContainer" containerID="ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08"
Jul 30 11:43:06.622732 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:43:06.622714    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08\": container with ID starting with ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08 not found: ID does not exist" containerID="ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08"
Jul 30 11:43:06.622779 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.622744    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08"} err="failed to get container status \"ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08\": rpc error: code = NotFound desc = could not find container \"ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08\": container with ID starting with ca72af67c124aa9c6d201eaad22f0c85a4ce6ed1c1f2fc3d790b3b25352e1c08 not found: ID does not exist"
Jul 30 11:43:06.648577 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.648551    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"]
Jul 30 11:43:06.652249 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.652226    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-060e9f/rate-test-model-060e9f-kserve-57657549f5-px8bj"]
Jul 30 11:43:06.688993 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.688972    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:06.688993 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.688992    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:06.689118 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.689001    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8knv7\" (UniqueName: \"kubernetes.io/projected/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-kube-api-access-8knv7\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:06.689118 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.689014    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:06.855492 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:06.855419    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" path="/var/lib/kubelet/pods/e4b0f1ee-117d-4fd7-b65c-1e745007a8e9/volumes"
Jul 30 11:43:07.109915 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.109816    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz" podUID="ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" containerName="main" containerID="cri-o://fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70" gracePeriod=45
Jul 30 11:43:07.369854 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.369804    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:43:07.395381 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.395352    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-dshm\") pod \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") "
Jul 30 11:43:07.395494 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.395402    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k2hq\" (UniqueName: \"kubernetes.io/projected/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-kube-api-access-6k2hq\") pod \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") "
Jul 30 11:43:07.395494 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.395457    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-home\") pod \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") "
Jul 30 11:43:07.395494 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.395482    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tmp-dir\") pod \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") "
Jul 30 11:43:07.395664 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.395512    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-model-cache\") pod \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") "
Jul 30 11:43:07.395664 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.395541    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tls-certs\") pod \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\" (UID: \"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5\") "
Jul 30 11:43:07.395770 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.395683    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-home" (OuterVolumeSpecName: "home") pod "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" (UID: "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:43:07.395824 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.395812    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:07.395956 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.395899    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" (UID: "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:43:07.395956 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.395944    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" (UID: "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:43:07.397468 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.397445    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-dshm" (OuterVolumeSpecName: "dshm") pod "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" (UID: "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:43:07.397646 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.397628    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" (UID: "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:43:07.397896 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.397881    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-kube-api-access-6k2hq" (OuterVolumeSpecName: "kube-api-access-6k2hq") pod "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" (UID: "ee09f3fd-24c0-4f9b-a970-b41e33ab12c5"). InnerVolumeSpecName "kube-api-access-6k2hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:43:07.497059 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.497030    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:07.497059 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.497058    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k2hq\" (UniqueName: \"kubernetes.io/projected/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-kube-api-access-6k2hq\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:07.497253 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.497073    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:07.497253 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.497086    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:07.497253 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.497118    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:07.622617 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.622538    2571 generic.go:356] "Generic (PLEG): container finished" podID="ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" containerID="fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70" exitCode=0
Jul 30 11:43:07.622617 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.622593    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"
Jul 30 11:43:07.622793 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.622628    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz" event={"ID":"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5","Type":"ContainerDied","Data":"fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70"}
Jul 30 11:43:07.622793 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.622670    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz" event={"ID":"ee09f3fd-24c0-4f9b-a970-b41e33ab12c5","Type":"ContainerDied","Data":"1963f2d341009b9d53636d28db469025d0bea373ff8cd72e6d68ecbc2d62d3de"}
Jul 30 11:43:07.622793 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.622686    2571 scope.go:117] "RemoveContainer" containerID="fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70"
Jul 30 11:43:07.632400 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.632383    2571 scope.go:117] "RemoveContainer" containerID="fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70"
Jul 30 11:43:07.632699 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:43:07.632677    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70\": container with ID starting with fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70 not found: ID does not exist" containerID="fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70"
Jul 30 11:43:07.632770 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.632705    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70"} err="failed to get container status \"fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70\": rpc error: code = NotFound desc = could not find container \"fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70\": container with ID starting with fa5df8e8b7805cc1901abb47a0075db53eb84b3277a94d3b94e283b0bfaf0e70 not found: ID does not exist"
Jul 30 11:43:07.646570 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.646547    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"]
Jul 30 11:43:07.651796 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.651769    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-945db9/rate-test-model-945db9-kserve-6c9fffb6c-fbdcz"]
Jul 30 11:43:07.744345 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.744322    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:07.765084 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.765057    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7382-139810755936304_667989e1-6f19-4ce3-9371-7697df061872/test-curl-7382-139810755936304/0.log"
Jul 30 11:43:07.786050 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.786022    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7382-139810755936304"]
Jul 30 11:43:07.789626 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.789603    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7382-139810755936304"]
Jul 30 11:43:07.800561 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.800543    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dqg5\" (UniqueName: \"kubernetes.io/projected/667989e1-6f19-4ce3-9371-7697df061872-kube-api-access-8dqg5\") pod \"667989e1-6f19-4ce3-9371-7697df061872\" (UID: \"667989e1-6f19-4ce3-9371-7697df061872\") "
Jul 30 11:43:07.802557 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.802528    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667989e1-6f19-4ce3-9371-7697df061872-kube-api-access-8dqg5" (OuterVolumeSpecName: "kube-api-access-8dqg5") pod "667989e1-6f19-4ce3-9371-7697df061872" (UID: "667989e1-6f19-4ce3-9371-7697df061872"). InnerVolumeSpecName "kube-api-access-8dqg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:43:07.901792 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.901724    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dqg5\" (UniqueName: \"kubernetes.io/projected/667989e1-6f19-4ce3-9371-7697df061872-kube-api-access-8dqg5\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:07.911965 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.911938    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7382-139810755936304"]
Jul 30 11:43:07.912292 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.912274    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667989e1-6f19-4ce3-9371-7697df061872" containerName="test-curl-7382-139810755936304"
Jul 30 11:43:07.912378 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.912294    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="667989e1-6f19-4ce3-9371-7697df061872" containerName="test-curl-7382-139810755936304"
Jul 30 11:43:07.912378 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.912309    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" containerName="main"
Jul 30 11:43:07.912378 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.912317    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" containerName="main"
Jul 30 11:43:07.912378 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.912339    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" containerName="main"
Jul 30 11:43:07.912378 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.912347    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" containerName="main"
Jul 30 11:43:07.912617 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.912421    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b0f1ee-117d-4fd7-b65c-1e745007a8e9" containerName="main"
Jul 30 11:43:07.912617 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.912432    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="667989e1-6f19-4ce3-9371-7697df061872" containerName="test-curl-7382-139810755936304"
Jul 30 11:43:07.912617 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.912442    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" containerName="main"
Jul 30 11:43:07.915352 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.915332    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:07.921834 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:07.921811    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810755936304"]
Jul 30 11:43:08.002442 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.002411    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9dvj\" (UniqueName: \"kubernetes.io/projected/18c60cbb-9773-435a-87f2-406b5d2d8bd0-kube-api-access-h9dvj\") pod \"test-curl-7382-139810755936304\" (UID: \"18c60cbb-9773-435a-87f2-406b5d2d8bd0\") " pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:08.103307 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.103281    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9dvj\" (UniqueName: \"kubernetes.io/projected/18c60cbb-9773-435a-87f2-406b5d2d8bd0-kube-api-access-h9dvj\") pod \"test-curl-7382-139810755936304\" (UID: \"18c60cbb-9773-435a-87f2-406b5d2d8bd0\") " pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:08.112493 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.112466    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9dvj\" (UniqueName: \"kubernetes.io/projected/18c60cbb-9773-435a-87f2-406b5d2d8bd0-kube-api-access-h9dvj\") pod \"test-curl-7382-139810755936304\" (UID: \"18c60cbb-9773-435a-87f2-406b5d2d8bd0\") " pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:08.224168 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.224138    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:08.578732 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.578705    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7382-139810755936304"]
Jul 30 11:43:08.580844 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:43:08.580816    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18c60cbb_9773_435a_87f2_406b5d2d8bd0.slice/crio-3c3b2d7aa97b29734e36c90b0448827fbb85fdd2ab626826fa18c0df4c015240 WatchSource:0}: Error finding container 3c3b2d7aa97b29734e36c90b0448827fbb85fdd2ab626826fa18c0df4c015240: Status 404 returned error can't find the container with id 3c3b2d7aa97b29734e36c90b0448827fbb85fdd2ab626826fa18c0df4c015240
Jul 30 11:43:08.627571 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.627531    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810755936304" event={"ID":"18c60cbb-9773-435a-87f2-406b5d2d8bd0","Type":"ContainerStarted","Data":"3c3b2d7aa97b29734e36c90b0448827fbb85fdd2ab626826fa18c0df4c015240"}
Jul 30 11:43:08.629415 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.629397    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09e5af9cbbfff0af2aa162298f010f80b5cbbe8d3d5931aee0827f44dd7f518f"
Jul 30 11:43:08.629503 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.629414    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:08.640410 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.640385    2571 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7382-139810755936304" oldPodUID="667989e1-6f19-4ce3-9371-7697df061872" podUID="18c60cbb-9773-435a-87f2-406b5d2d8bd0"
Jul 30 11:43:08.854660 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.854589    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667989e1-6f19-4ce3-9371-7697df061872" path="/var/lib/kubelet/pods/667989e1-6f19-4ce3-9371-7697df061872/volumes"
Jul 30 11:43:08.854953 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:08.854941    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee09f3fd-24c0-4f9b-a970-b41e33ab12c5" path="/var/lib/kubelet/pods/ee09f3fd-24c0-4f9b-a970-b41e33ab12c5/volumes"
Jul 30 11:43:09.635244 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:09.635207    2571 generic.go:356] "Generic (PLEG): container finished" podID="18c60cbb-9773-435a-87f2-406b5d2d8bd0" containerID="86f893098dfd8c9d72ba7c1659db29e6661823d1bbd0c3eec16a5b089c02e401" exitCode=0
Jul 30 11:43:09.635596 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:09.635289    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7382-139810755936304" event={"ID":"18c60cbb-9773-435a-87f2-406b5d2d8bd0","Type":"ContainerDied","Data":"86f893098dfd8c9d72ba7c1659db29e6661823d1bbd0c3eec16a5b089c02e401"}
Jul 30 11:43:10.763002 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:10.762977    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:10.785086 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:10.785064    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7382-139810755936304_18c60cbb-9773-435a-87f2-406b5d2d8bd0/test-curl-7382-139810755936304/0.log"
Jul 30 11:43:10.806328 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:10.806306    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7382-139810755936304"]
Jul 30 11:43:10.810314 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:10.810293    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7382-139810755936304"]
Jul 30 11:43:10.822086 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:10.822064    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9dvj\" (UniqueName: \"kubernetes.io/projected/18c60cbb-9773-435a-87f2-406b5d2d8bd0-kube-api-access-h9dvj\") pod \"18c60cbb-9773-435a-87f2-406b5d2d8bd0\" (UID: \"18c60cbb-9773-435a-87f2-406b5d2d8bd0\") "
Jul 30 11:43:10.823989 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:10.823967    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c60cbb-9773-435a-87f2-406b5d2d8bd0-kube-api-access-h9dvj" (OuterVolumeSpecName: "kube-api-access-h9dvj") pod "18c60cbb-9773-435a-87f2-406b5d2d8bd0" (UID: "18c60cbb-9773-435a-87f2-406b5d2d8bd0"). InnerVolumeSpecName "kube-api-access-h9dvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:43:10.853942 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:10.853917    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18c60cbb-9773-435a-87f2-406b5d2d8bd0" path="/var/lib/kubelet/pods/18c60cbb-9773-435a-87f2-406b5d2d8bd0/volumes"
Jul 30 11:43:10.922528 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:10.922472    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9dvj\" (UniqueName: \"kubernetes.io/projected/18c60cbb-9773-435a-87f2-406b5d2d8bd0-kube-api-access-h9dvj\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:43:11.617173 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.617132    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"]
Jul 30 11:43:11.617432 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.617420    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c60cbb-9773-435a-87f2-406b5d2d8bd0" containerName="test-curl-7382-139810755936304"
Jul 30 11:43:11.617481 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.617434    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c60cbb-9773-435a-87f2-406b5d2d8bd0" containerName="test-curl-7382-139810755936304"
Jul 30 11:43:11.617513 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.617497    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="18c60cbb-9773-435a-87f2-406b5d2d8bd0" containerName="test-curl-7382-139810755936304"
Jul 30 11:43:11.620462 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.620445    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.623032 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.623009    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d722d1-openshift-default-dockercfg-sgp95\""
Jul 30 11:43:11.623245 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.623011    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 30 11:43:11.632071 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.632045    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"]
Jul 30 11:43:11.644577 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.644552    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7382-139810755936304"
Jul 30 11:43:11.644577 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.644569    2571 scope.go:117] "RemoveContainer" containerID="86f893098dfd8c9d72ba7c1659db29e6661823d1bbd0c3eec16a5b089c02e401"
Jul 30 11:43:11.728111 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.728078    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-credential-socket\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.728229 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.728144    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-envoy\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.728229 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.728171    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2b5666-c22c-4672-971e-842348f476c6-istiod-ca-cert\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.728229 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.728193    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4krg\" (UniqueName: \"kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-kube-api-access-j4krg\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.728351 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.728292    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-data\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.728351 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.728327    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-socket\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.728351 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.728345    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2b5666-c22c-4672-971e-842348f476c6-istio-podinfo\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.728452 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.728369    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-certs\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.728452 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.728388    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-istio-token\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829320 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829290    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-socket\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829320 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829321    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2b5666-c22c-4672-971e-842348f476c6-istio-podinfo\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829787 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829340    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-certs\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829787 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829361    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-istio-token\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829787 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829496    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-credential-socket\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829787 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829538    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-envoy\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829787 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2b5666-c22c-4672-971e-842348f476c6-istiod-ca-cert\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829787 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829606    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4krg\" (UniqueName: \"kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-kube-api-access-j4krg\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829787 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-data\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829787 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-socket\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.829787 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829743    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-certs\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.830168 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829827    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-credential-socket\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.830168 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.829982    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-data\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.830404 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.830380    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2b5666-c22c-4672-971e-842348f476c6-istiod-ca-cert\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.831871 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.831851    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-envoy\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.832114 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.832080    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2b5666-c22c-4672-971e-842348f476c6-istio-podinfo\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.837598 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.837570    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-istio-token\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.837778 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.837758    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4krg\" (UniqueName: \"kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-kube-api-access-j4krg\") pod \"e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") " pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:11.930362 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:11.930339    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:12.060995 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:12.060965    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"]
Jul 30 11:43:12.062615 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:43:12.062586    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2b5666_c22c_4672_971e_842348f476c6.slice/crio-ad5e8fae018ec9e08ccd85a0408a2b88b3ce38a5846d827a286243249ecd2d82 WatchSource:0}: Error finding container ad5e8fae018ec9e08ccd85a0408a2b88b3ce38a5846d827a286243249ecd2d82: Status 404 returned error can't find the container with id ad5e8fae018ec9e08ccd85a0408a2b88b3ce38a5846d827a286243249ecd2d82
Jul 30 11:43:12.540208 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:12.540172    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:12.639878 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:12.639842    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:12.650235 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:12.650202    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" event={"ID":"8f2b5666-c22c-4672-971e-842348f476c6","Type":"ContainerStarted","Data":"ad5e8fae018ec9e08ccd85a0408a2b88b3ce38a5846d827a286243249ecd2d82"}
Jul 30 11:43:15.729420 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:15.729382    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:43:15.729699 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:15.729459    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:43:15.729699 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:15.729488    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:43:16.669684 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:16.669650    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" event={"ID":"8f2b5666-c22c-4672-971e-842348f476c6","Type":"ContainerStarted","Data":"8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b"}
Jul 30 11:43:16.691252 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:16.691182    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" podStartSLOduration=2.027160018 podStartE2EDuration="5.691167413s" totalImagesPullingTime="3.664007395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:43:16.689739431 +0000 UTC m=+3302.410623692" watchObservedRunningTime="2026-07-30 11:43:16.691167413 +0000 UTC m=+3302.412051731"
Jul 30 11:43:16.931172 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:16.931143    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:16.936123 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:16.936081    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:17.674364 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:17.674326    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:17.675441 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:17.675420    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:18.630922 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:18.630885    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:18.730868 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:18.730835    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:19.766002 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:19.765963    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:19.904019 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:19.903986    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:20.238852 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.238818    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:20.338899 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.338861    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:20.366155 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.366124    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"]
Jul 30 11:43:20.369191 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.369174    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:43:20.371920 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.371886    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d722d1\""
Jul 30 11:43:20.372038 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.371934    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d722d1-dockercfg-pcwz8\""
Jul 30 11:43:20.379625 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.379599    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"]
Jul 30 11:43:20.380858 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.380841    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"]
Jul 30 11:43:20.383767 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.383751    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:43:20.393156 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.393136    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"]
Jul 30 11:43:20.506024 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.505949    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9\" (UID: \"f58bf83c-ac2c-4064-98ac-6a6761a68b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:43:20.506024 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.505990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xmqv\" (UniqueName: \"kubernetes.io/projected/971eae80-fd38-4164-8717-ade4cf3d5c8a-kube-api-access-7xmqv\") pod \"payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97\" (UID: \"971eae80-fd38-4164-8717-ade4cf3d5c8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:43:20.506222 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.506023    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971eae80-fd38-4164-8717-ade4cf3d5c8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97\" (UID: \"971eae80-fd38-4164-8717-ade4cf3d5c8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:43:20.506222 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.506070    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw9fd\" (UniqueName: \"kubernetes.io/projected/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-kube-api-access-fw9fd\") pod \"payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9\" (UID: \"f58bf83c-ac2c-4064-98ac-6a6761a68b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:43:20.607014 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.606981    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9\" (UID: \"f58bf83c-ac2c-4064-98ac-6a6761a68b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:43:20.607176 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.607019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xmqv\" (UniqueName: \"kubernetes.io/projected/971eae80-fd38-4164-8717-ade4cf3d5c8a-kube-api-access-7xmqv\") pod \"payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97\" (UID: \"971eae80-fd38-4164-8717-ade4cf3d5c8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:43:20.607176 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.607040    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971eae80-fd38-4164-8717-ade4cf3d5c8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97\" (UID: \"971eae80-fd38-4164-8717-ade4cf3d5c8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:43:20.607176 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.607083    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9fd\" (UniqueName: \"kubernetes.io/projected/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-kube-api-access-fw9fd\") pod \"payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9\" (UID: \"f58bf83c-ac2c-4064-98ac-6a6761a68b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:43:20.607793 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.607772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971eae80-fd38-4164-8717-ade4cf3d5c8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97\" (UID: \"971eae80-fd38-4164-8717-ade4cf3d5c8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:43:20.607844 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.607783    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9\" (UID: \"f58bf83c-ac2c-4064-98ac-6a6761a68b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:43:20.615844 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.615815    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9fd\" (UniqueName: \"kubernetes.io/projected/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-kube-api-access-fw9fd\") pod \"payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9\" (UID: \"f58bf83c-ac2c-4064-98ac-6a6761a68b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:43:20.616281 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.616260    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xmqv\" (UniqueName: \"kubernetes.io/projected/971eae80-fd38-4164-8717-ade4cf3d5c8a-kube-api-access-7xmqv\") pod \"payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97\" (UID: \"971eae80-fd38-4164-8717-ade4cf3d5c8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:43:20.680271 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.680237    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:43:20.693063 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.693033    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:43:20.825419 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.825292    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"]
Jul 30 11:43:20.828523 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:43:20.828488    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58bf83c_ac2c_4064_98ac_6a6761a68b3b.slice/crio-724b0e6feaa77879b6440256c4170f60de49f34816a9312b2c932e9de2063b44 WatchSource:0}: Error finding container 724b0e6feaa77879b6440256c4170f60de49f34816a9312b2c932e9de2063b44: Status 404 returned error can't find the container with id 724b0e6feaa77879b6440256c4170f60de49f34816a9312b2c932e9de2063b44
Jul 30 11:43:20.856218 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:43:20.856188    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971eae80_fd38_4164_8717_ade4cf3d5c8a.slice/crio-f949270adf9799e88783f9efbbc8f702e07fb1d1831ac3ae0c97137a5504e8ff WatchSource:0}: Error finding container f949270adf9799e88783f9efbbc8f702e07fb1d1831ac3ae0c97137a5504e8ff: Status 404 returned error can't find the container with id f949270adf9799e88783f9efbbc8f702e07fb1d1831ac3ae0c97137a5504e8ff
Jul 30 11:43:20.856426 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:20.856408    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"]
Jul 30 11:43:21.697023 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:21.696990    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97" event={"ID":"971eae80-fd38-4164-8717-ade4cf3d5c8a","Type":"ContainerStarted","Data":"c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9"}
Jul 30 11:43:21.697240 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:21.697049    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97" event={"ID":"971eae80-fd38-4164-8717-ade4cf3d5c8a","Type":"ContainerStarted","Data":"f949270adf9799e88783f9efbbc8f702e07fb1d1831ac3ae0c97137a5504e8ff"}
Jul 30 11:43:21.697240 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:21.697073    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:43:21.698514 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:21.698490    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9" event={"ID":"f58bf83c-ac2c-4064-98ac-6a6761a68b3b","Type":"ContainerStarted","Data":"16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963"}
Jul 30 11:43:21.698575 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:21.698533    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9" event={"ID":"f58bf83c-ac2c-4064-98ac-6a6761a68b3b","Type":"ContainerStarted","Data":"724b0e6feaa77879b6440256c4170f60de49f34816a9312b2c932e9de2063b44"}
Jul 30 11:43:21.698657 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:21.698615    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:43:21.714713 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:21.714668    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97" podStartSLOduration=1.714655564 podStartE2EDuration="1.714655564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:43:21.714273127 +0000 UTC m=+3307.435157387" watchObservedRunningTime="2026-07-30 11:43:21.714655564 +0000 UTC m=+3307.435539802"
Jul 30 11:43:21.730149 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:21.730111    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9" podStartSLOduration=1.730084834 podStartE2EDuration="1.730084834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:43:21.728970491 +0000 UTC m=+3307.449854762" watchObservedRunningTime="2026-07-30 11:43:21.730084834 +0000 UTC m=+3307.450969095"
Jul 30 11:43:22.384405 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:22.384374    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:43:32.675681 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:32.675637    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 30 11:43:32.676110 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:32.675710    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 30 11:43:32.704006 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:32.703980    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:43:32.704166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:32.704151    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:43:32.747398 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:32.747362    2571 generic.go:356] "Generic (PLEG): container finished" podID="8f2b5666-c22c-4672-971e-842348f476c6" containerID="8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b" exitCode=0
Jul 30 11:43:32.747576 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:32.747435    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" event={"ID":"8f2b5666-c22c-4672-971e-842348f476c6","Type":"ContainerDied","Data":"8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b"}
Jul 30 11:43:32.747873 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:32.747855    2571 scope.go:117] "RemoveContainer" containerID="8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b"
Jul 30 11:43:32.750199 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:32.750171    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:43:32.750292 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:32.750244    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:43:32.750357 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:32.750294    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:43:33.756599 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:33.756563    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" event={"ID":"8f2b5666-c22c-4672-971e-842348f476c6","Type":"ContainerStarted","Data":"8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2"}
Jul 30 11:43:33.931664 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:33.931628    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:33.931960 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:33.931668    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 30 11:43:33.932157 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:33.932131    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 30 11:43:34.930821 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:34.930785    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 30 11:43:34.931294 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:34.930859    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 30 11:43:35.931221 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:35.931183    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 30 11:43:35.931791 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:35.931755    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 30 11:43:36.931536 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:36.931497    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 30 11:43:36.932175 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:36.932142    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 30 11:43:37.931734 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:37.931686    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 30 11:43:37.932280 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:37.931767    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 30 11:43:38.934789 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:38.934756    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:38.935208 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:38.935019    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:38.935838 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:38.935813    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:43:43.657192 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:43.657152    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:44.346623 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:44.346570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:44.737954 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:44.737919    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:44.846329 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:44.846294    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:46.750056 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:46.750013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:48.212769 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:48.212732    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:48.313127 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:48.313079    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:50.495525 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:50.495488    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:50.959369 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:50.959342    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:51.067128 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.067072    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:51.073010 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.072984    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"]
Jul 30 11:43:51.081212 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.081190    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:43:51.084491 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.084360    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"]
Jul 30 11:43:51.084614 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.084538    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f0ce38-dockercfg-tc2s5\""
Jul 30 11:43:51.084686 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.084538    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f0ce38\""
Jul 30 11:43:51.088283 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.088262    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"]
Jul 30 11:43:51.088396 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.088379    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:43:51.097479 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.097454    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"]
Jul 30 11:43:51.157748 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.157718    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhsn\" (UniqueName: \"kubernetes.io/projected/d074effb-916b-4917-8a25-7d59ec3d12c5-kube-api-access-sbhsn\") pod \"payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w\" (UID: \"d074effb-916b-4917-8a25-7d59ec3d12c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:43:51.157892 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.157755    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8p7g\" (UniqueName: \"kubernetes.io/projected/13f67b57-4dd3-4608-b068-3300925414ee-kube-api-access-z8p7g\") pod \"payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8\" (UID: \"13f67b57-4dd3-4608-b068-3300925414ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:43:51.157892 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.157774    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d074effb-916b-4917-8a25-7d59ec3d12c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w\" (UID: \"d074effb-916b-4917-8a25-7d59ec3d12c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:43:51.157892 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.157876    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f67b57-4dd3-4608-b068-3300925414ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8\" (UID: \"13f67b57-4dd3-4608-b068-3300925414ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:43:51.258504 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.258415    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f67b57-4dd3-4608-b068-3300925414ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8\" (UID: \"13f67b57-4dd3-4608-b068-3300925414ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:43:51.258504 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.258461    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhsn\" (UniqueName: \"kubernetes.io/projected/d074effb-916b-4917-8a25-7d59ec3d12c5-kube-api-access-sbhsn\") pod \"payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w\" (UID: \"d074effb-916b-4917-8a25-7d59ec3d12c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:43:51.258728 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.258503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8p7g\" (UniqueName: \"kubernetes.io/projected/13f67b57-4dd3-4608-b068-3300925414ee-kube-api-access-z8p7g\") pod \"payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8\" (UID: \"13f67b57-4dd3-4608-b068-3300925414ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:43:51.258728 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.258532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d074effb-916b-4917-8a25-7d59ec3d12c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w\" (UID: \"d074effb-916b-4917-8a25-7d59ec3d12c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:43:51.259416 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.259389    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f67b57-4dd3-4608-b068-3300925414ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8\" (UID: \"13f67b57-4dd3-4608-b068-3300925414ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:43:51.259518 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.259389    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d074effb-916b-4917-8a25-7d59ec3d12c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w\" (UID: \"d074effb-916b-4917-8a25-7d59ec3d12c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:43:51.268602 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.268575    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhsn\" (UniqueName: \"kubernetes.io/projected/d074effb-916b-4917-8a25-7d59ec3d12c5-kube-api-access-sbhsn\") pod \"payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w\" (UID: \"d074effb-916b-4917-8a25-7d59ec3d12c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:43:51.268735 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.268631    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8p7g\" (UniqueName: \"kubernetes.io/projected/13f67b57-4dd3-4608-b068-3300925414ee-kube-api-access-z8p7g\") pod \"payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8\" (UID: \"13f67b57-4dd3-4608-b068-3300925414ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:43:51.393012 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.392979    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:43:51.401518 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.401498    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:43:51.556153 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.556062    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"]
Jul 30 11:43:51.562069 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:43:51.562041    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd074effb_916b_4917_8a25_7d59ec3d12c5.slice/crio-c7fee43c7c09c068fb80992942d2f14a5201bfe3bf17d25563d05175bc4564bf WatchSource:0}: Error finding container c7fee43c7c09c068fb80992942d2f14a5201bfe3bf17d25563d05175bc4564bf: Status 404 returned error can't find the container with id c7fee43c7c09c068fb80992942d2f14a5201bfe3bf17d25563d05175bc4564bf
Jul 30 11:43:51.567791 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.567769    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"]
Jul 30 11:43:51.569089 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:43:51.569068    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f67b57_4dd3_4608_b068_3300925414ee.slice/crio-d751b09def52ff0cddb2816cdd008f271c9c922b2597bcca56cd488a9a421754 WatchSource:0}: Error finding container d751b09def52ff0cddb2816cdd008f271c9c922b2597bcca56cd488a9a421754: Status 404 returned error can't find the container with id d751b09def52ff0cddb2816cdd008f271c9c922b2597bcca56cd488a9a421754
Jul 30 11:43:51.839828 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.839721    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w" event={"ID":"d074effb-916b-4917-8a25-7d59ec3d12c5","Type":"ContainerStarted","Data":"43eb7f0f2d4adf25a9bcc34e1f739df2f99bb9ee9321f70e8b115dc2434dfab1"}
Jul 30 11:43:51.839828 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.839791    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w" event={"ID":"d074effb-916b-4917-8a25-7d59ec3d12c5","Type":"ContainerStarted","Data":"c7fee43c7c09c068fb80992942d2f14a5201bfe3bf17d25563d05175bc4564bf"}
Jul 30 11:43:51.839828 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.839821    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:43:51.841406 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.841380    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8" event={"ID":"13f67b57-4dd3-4608-b068-3300925414ee","Type":"ContainerStarted","Data":"0940f006dc34ed19de16842ffb0960fbc677416fb6632fa38560362de396dbda"}
Jul 30 11:43:51.841536 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.841427    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8" event={"ID":"13f67b57-4dd3-4608-b068-3300925414ee","Type":"ContainerStarted","Data":"d751b09def52ff0cddb2816cdd008f271c9c922b2597bcca56cd488a9a421754"}
Jul 30 11:43:51.841536 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.841454    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:43:51.861564 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.861507    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w" podStartSLOduration=0.861490326 podStartE2EDuration="861.490326ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:43:51.861309258 +0000 UTC m=+3337.582193520" watchObservedRunningTime="2026-07-30 11:43:51.861490326 +0000 UTC m=+3337.582374589"
Jul 30 11:43:51.879651 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:51.879602    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8" podStartSLOduration=0.879584324 podStartE2EDuration="879.584324ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:43:51.878752569 +0000 UTC m=+3337.599636831" watchObservedRunningTime="2026-07-30 11:43:51.879584324 +0000 UTC m=+3337.600468592"
Jul 30 11:43:52.343151 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:52.343109    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:43:53.197293 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:53.197256    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:43:58.066944 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:43:58.066906    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:44:02.848020 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:02.847990    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:44:02.848419 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:02.848061    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:44:02.951345 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:02.951314    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:44:10.016829 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:10.016795    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8_13f67b57-4dd3-4608-b068-3300925414ee/payload-processing/0.log"
Jul 30 11:44:18.377898 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.377859    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"]
Jul 30 11:44:18.381549 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.381528    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.385580 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.385554    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d722d1\"/\"default-dockercfg-xkqk8\""
Jul 30 11:44:18.385580 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.385573    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d722d1\"/\"kube-root-ca.crt\""
Jul 30 11:44:18.385735 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.385611    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d722d1\"/\"ipp-route-d722d1-kserve-self-signed-certs\""
Jul 30 11:44:18.385735 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.385612    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d722d1\"/\"openshift-service-ca.crt\""
Jul 30 11:44:18.389227 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.389206    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"]
Jul 30 11:44:18.400653 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.400626    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:44:18.479274 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.479243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-dshm\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.479403 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.479279    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79e5b28-863b-47fe-93b2-b027fc0adda2-tls-certs\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.479403 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.479299    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7rv5\" (UniqueName: \"kubernetes.io/projected/a79e5b28-863b-47fe-93b2-b027fc0adda2-kube-api-access-c7rv5\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.479553 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.479407    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-home\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.479553 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.479434    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-model-cache\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.479553 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.479467    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-tmp-dir\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.580297 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.580268    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-home\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.580297 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.580298    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-model-cache\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.580486 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.580324    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-tmp-dir\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.580486 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.580357    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-dshm\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.580486 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.580383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79e5b28-863b-47fe-93b2-b027fc0adda2-tls-certs\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.580486 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.580410    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7rv5\" (UniqueName: \"kubernetes.io/projected/a79e5b28-863b-47fe-93b2-b027fc0adda2-kube-api-access-c7rv5\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.580688 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.580665    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-tmp-dir\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.580761 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.580737    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-home\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.580808 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.580786    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-model-cache\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.582589 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.582565    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-dshm\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.582847 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.582826    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79e5b28-863b-47fe-93b2-b027fc0adda2-tls-certs\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.588319 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.588296    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7rv5\" (UniqueName: \"kubernetes.io/projected/a79e5b28-863b-47fe-93b2-b027fc0adda2-kube-api-access-c7rv5\") pod \"ipp-route-d722d1-kserve-56b44b8d55-mvpvm\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") " pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.692461 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.692429    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:18.822881 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.822859    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"]
Jul 30 11:44:18.824959 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:44:18.824933    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda79e5b28_863b_47fe_93b2_b027fc0adda2.slice/crio-9765d41894ad9159a73a9400cd4cc2e00f4d2adcbfd18ef3ed86d08f5037d898 WatchSource:0}: Error finding container 9765d41894ad9159a73a9400cd4cc2e00f4d2adcbfd18ef3ed86d08f5037d898: Status 404 returned error can't find the container with id 9765d41894ad9159a73a9400cd4cc2e00f4d2adcbfd18ef3ed86d08f5037d898
Jul 30 11:44:18.955901 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.955809    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm" event={"ID":"a79e5b28-863b-47fe-93b2-b027fc0adda2","Type":"ContainerStarted","Data":"9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320"}
Jul 30 11:44:18.955901 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.955852    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm" event={"ID":"a79e5b28-863b-47fe-93b2-b027fc0adda2","Type":"ContainerStarted","Data":"9765d41894ad9159a73a9400cd4cc2e00f4d2adcbfd18ef3ed86d08f5037d898"}
Jul 30 11:44:18.975741 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:18.975691    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm" podStartSLOduration=0.975676155 podStartE2EDuration="975.676155ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:44:18.973999364 +0000 UTC m=+3364.694883624" watchObservedRunningTime="2026-07-30 11:44:18.975676155 +0000 UTC m=+3364.696560416"
Jul 30 11:44:21.094475 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:21.094439    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:44:28.693024 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:28.692991    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:28.693024 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:28.693034    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:28.705367 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:28.705344    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:29.009845 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:29.009754    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:44:59.194885 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:44:59.194841    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:45:01.627131 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:01.627081    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:10.215303 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:10.215269    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97_971eae80-fd38-4164-8717-ade4cf3d5c8a/payload-processing/0.log"
Jul 30 11:45:10.462990 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:10.462962    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8_13f67b57-4dd3-4608-b068-3300925414ee/payload-processing/0.log"
Jul 30 11:45:10.759686 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:10.759649    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:11.176412 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.176329    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"]
Jul 30 11:45:11.179646 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.179629    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.182557 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.182536    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-a93cc2-openshift-default-dockercfg-rrkk7\""
Jul 30 11:45:11.193158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.193130    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"]
Jul 30 11:45:11.207077 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.207052    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:11.296369 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.296337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-data\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.296369 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.296371    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-certs\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.296777 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.296503    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-envoy\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.296777 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.296558    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-socket\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.296777 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.296588    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.296777 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.296633    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-token\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.296777 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.296676    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-credential-socket\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.296777 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.296734    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-podinfo\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.296777 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.296759    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c949n\" (UniqueName: \"kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-kube-api-access-c949n\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398131 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398082    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-data\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398131 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-certs\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398327 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398187    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-envoy\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398327 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398215    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-socket\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398401 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398456 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398400    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-token\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398451    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-credential-socket\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398491    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-podinfo\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398617 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398516    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c949n\" (UniqueName: \"kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-kube-api-access-c949n\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398617 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398528    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-data\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398617 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398551    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-socket\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398617 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398576    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-certs\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.398835 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.398816    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-credential-socket\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.399133 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.399093    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.400559 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.400530    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-envoy\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.400955 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.400935    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-podinfo\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.406872 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.406850    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-token\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.407279 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.407256    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c949n\" (UniqueName: \"kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-kube-api-access-c949n\") pod \"e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") " pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.464294 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.464271    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:11.491266 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.491239    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:11.628546 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.628523    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"]
Jul 30 11:45:11.630673 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:45:11.630643    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6039dbd6_58d1_4ab9_8f98_9940b7c56406.slice/crio-88d22b6fa4ae538d657a3118eaa4d70021f1a9af3350ffdf8bce4b6231526c54 WatchSource:0}: Error finding container 88d22b6fa4ae538d657a3118eaa4d70021f1a9af3350ffdf8bce4b6231526c54: Status 404 returned error can't find the container with id 88d22b6fa4ae538d657a3118eaa4d70021f1a9af3350ffdf8bce4b6231526c54
Jul 30 11:45:11.633411 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.633380    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:45:11.633476 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.633453    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:45:11.633514 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:11.633481    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:45:12.193931 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:12.193892    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" event={"ID":"6039dbd6-58d1-4ab9-8f98-9940b7c56406","Type":"ContainerStarted","Data":"ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e"}
Jul 30 11:45:12.194258 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:12.193949    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" event={"ID":"6039dbd6-58d1-4ab9-8f98-9940b7c56406","Type":"ContainerStarted","Data":"88d22b6fa4ae538d657a3118eaa4d70021f1a9af3350ffdf8bce4b6231526c54"}
Jul 30 11:45:12.225439 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:12.225394    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" podStartSLOduration=1.225379457 podStartE2EDuration="1.225379457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:45:12.223420146 +0000 UTC m=+3417.944304407" watchObservedRunningTime="2026-07-30 11:45:12.225379457 +0000 UTC m=+3417.946263718"
Jul 30 11:45:12.492341 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:12.492247    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:12.497439 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:12.497414    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:13.197994 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:13.197960    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:13.199021 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:13.198995    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:14.264473 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:14.264433    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:14.366783 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:14.366743    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:15.684155 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:15.684124    2571 scope.go:117] "RemoveContainer" containerID="993f994ffa23824fdbe13bb1ab44027544bb4563da3f67f0ce25441f5752ca2e"
Jul 30 11:45:16.435163 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:16.435109    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:16.535119 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:16.535067    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:16.916364 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:16.916282    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:17.021203 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:17.021169    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:21.120752 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:21.120719    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:45:28.981449 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:28.981387    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:32.294676 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:32.294644    2571 generic.go:356] "Generic (PLEG): container finished" podID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerID="ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e" exitCode=0
Jul 30 11:45:32.295119 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:32.294695    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" event={"ID":"6039dbd6-58d1-4ab9-8f98-9940b7c56406","Type":"ContainerDied","Data":"ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e"}
Jul 30 11:45:32.295119 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:32.295064    2571 scope.go:117] "RemoveContainer" containerID="ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e"
Jul 30 11:45:32.296323 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:32.296292    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:45:32.296395 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:32.296368    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:45:32.296442 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:32.296406    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 30 11:45:33.301579 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:33.301540    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" event={"ID":"6039dbd6-58d1-4ab9-8f98-9940b7c56406","Type":"ContainerStarted","Data":"18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4"}
Jul 30 11:45:33.492199 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:33.492167    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:33.492486 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:33.492463    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 30 11:45:33.492533 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:33.492513    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 30 11:45:34.492204 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:34.492163    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 30 11:45:34.492661 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:34.492228    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 30 11:45:35.492412 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:35.492373    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 30 11:45:35.492785 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:35.492431    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 30 11:45:36.040966 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:36.040911    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:36.142133 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:36.142084    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:36.492192 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:36.492153    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 30 11:45:36.492402 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:36.492213    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 30 11:45:37.491851 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:37.491815    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 30 11:45:37.492274 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:37.491875    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 30 11:45:38.492639 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:38.492601    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 30 11:45:38.493024 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:38.492658    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 30 11:45:38.908432 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:38.908356    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:39.025845 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:39.025810    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:39.123455 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:39.123422    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:39.495965 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:39.495936    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:39.496357 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:39.496220    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:39.496982 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:39.496966    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:39.503230 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:39.503206    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:41.914164 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:41.914126    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:45:46.197792 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:46.197756    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:46.297672 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:46.297636    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:46.468186 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:46.468075    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"]
Jul 30 11:45:46.468420 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:46.468364    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy" containerID="cri-o://18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4" gracePeriod=30
Jul 30 11:45:46.495931 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:46.495889    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:46.727882 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:46.727799    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:45:51.705550 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.705526    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:51.763886 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.763826    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-token\") pod \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") "
Jul 30 11:45:51.763886 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.763876    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-certs\") pod \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") "
Jul 30 11:45:51.764166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.763893    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-envoy\") pod \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") "
Jul 30 11:45:51.764166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.763922    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-data\") pod \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") "
Jul 30 11:45:51.764166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.763973    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istiod-ca-cert\") pod \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") "
Jul 30 11:45:51.764166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.763996    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-podinfo\") pod \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") "
Jul 30 11:45:51.764166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.764030    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-socket\") pod \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") "
Jul 30 11:45:51.764166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.764067    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c949n\" (UniqueName: \"kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-kube-api-access-c949n\") pod \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") "
Jul 30 11:45:51.764166 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.764139    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-credential-socket\") pod \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\" (UID: \"6039dbd6-58d1-4ab9-8f98-9940b7c56406\") "
Jul 30 11:45:51.764523 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.764191    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6039dbd6-58d1-4ab9-8f98-9940b7c56406" (UID: "6039dbd6-58d1-4ab9-8f98-9940b7c56406"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:45:51.764523 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.764388    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:45:51.764523 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.764397    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6039dbd6-58d1-4ab9-8f98-9940b7c56406" (UID: "6039dbd6-58d1-4ab9-8f98-9940b7c56406"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:45:51.764523 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.764437    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-data" (OuterVolumeSpecName: "istio-data") pod "6039dbd6-58d1-4ab9-8f98-9940b7c56406" (UID: "6039dbd6-58d1-4ab9-8f98-9940b7c56406"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:45:51.764523 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.764504    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6039dbd6-58d1-4ab9-8f98-9940b7c56406" (UID: "6039dbd6-58d1-4ab9-8f98-9940b7c56406"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:45:51.764803 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.764586    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6039dbd6-58d1-4ab9-8f98-9940b7c56406" (UID: "6039dbd6-58d1-4ab9-8f98-9940b7c56406"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:45:51.766332 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.766303    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-kube-api-access-c949n" (OuterVolumeSpecName: "kube-api-access-c949n") pod "6039dbd6-58d1-4ab9-8f98-9940b7c56406" (UID: "6039dbd6-58d1-4ab9-8f98-9940b7c56406"). InnerVolumeSpecName "kube-api-access-c949n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:45:51.766332 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.766324    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6039dbd6-58d1-4ab9-8f98-9940b7c56406" (UID: "6039dbd6-58d1-4ab9-8f98-9940b7c56406"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:45:51.766473 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.766336    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-token" (OuterVolumeSpecName: "istio-token") pod "6039dbd6-58d1-4ab9-8f98-9940b7c56406" (UID: "6039dbd6-58d1-4ab9-8f98-9940b7c56406"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:45:51.766473 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.766341    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6039dbd6-58d1-4ab9-8f98-9940b7c56406" (UID: "6039dbd6-58d1-4ab9-8f98-9940b7c56406"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 11:45:51.864749 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.864721    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-envoy\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:45:51.864749 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.864744    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-data\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:45:51.864749 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.864755    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istiod-ca-cert\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:45:51.864938 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.864763    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-podinfo\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:45:51.864938 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.864770    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-workload-socket\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:45:51.864938 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.864780    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c949n\" (UniqueName: \"kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-kube-api-access-c949n\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:45:51.864938 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.864788    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039dbd6-58d1-4ab9-8f98-9940b7c56406-credential-socket\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:45:51.864938 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:51.864796    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6039dbd6-58d1-4ab9-8f98-9940b7c56406-istio-token\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:45:52.387748 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.387702    2571 generic.go:356] "Generic (PLEG): container finished" podID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerID="18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4" exitCode=0
Jul 30 11:45:52.387961 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.387759    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"
Jul 30 11:45:52.387961 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.387789    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" event={"ID":"6039dbd6-58d1-4ab9-8f98-9940b7c56406","Type":"ContainerDied","Data":"18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4"}
Jul 30 11:45:52.387961 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.387840    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6" event={"ID":"6039dbd6-58d1-4ab9-8f98-9940b7c56406","Type":"ContainerDied","Data":"88d22b6fa4ae538d657a3118eaa4d70021f1a9af3350ffdf8bce4b6231526c54"}
Jul 30 11:45:52.387961 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.387864    2571 scope.go:117] "RemoveContainer" containerID="18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4"
Jul 30 11:45:52.396831 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.396814    2571 scope.go:117] "RemoveContainer" containerID="ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e"
Jul 30 11:45:52.404309 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.404292    2571 scope.go:117] "RemoveContainer" containerID="18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4"
Jul 30 11:45:52.404558 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:45:52.404542    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4\": container with ID starting with 18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4 not found: ID does not exist" containerID="18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4"
Jul 30 11:45:52.404595 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.404568    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4"} err="failed to get container status \"18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4\": rpc error: code = NotFound desc = could not find container \"18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4\": container with ID starting with 18a4b35a56615c3508c6c070df292c4a146be453508800434a17da8046b4aae4 not found: ID does not exist"
Jul 30 11:45:52.404595 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.404584    2571 scope.go:117] "RemoveContainer" containerID="ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e"
Jul 30 11:45:52.404802 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:45:52.404787    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e\": container with ID starting with ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e not found: ID does not exist" containerID="ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e"
Jul 30 11:45:52.404841 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.404807    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e"} err="failed to get container status \"ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e\": rpc error: code = NotFound desc = could not find container \"ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e\": container with ID starting with ad2643ba017eaa1467b3db88de76be115dee4ae57ef9937e7c090312a471d10e not found: ID does not exist"
Jul 30 11:45:52.409331 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.409307    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"]
Jul 30 11:45:52.415633 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.415612    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a93cc2-openshift-default-6d99bf95c8-pjsj6"]
Jul 30 11:45:52.855982 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:45:52.855952    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" path="/var/lib/kubelet/pods/6039dbd6-58d1-4ab9-8f98-9940b7c56406/volumes"
Jul 30 11:46:02.521507 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:02.521467    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:46:07.409777 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.409743    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"]
Jul 30 11:46:07.410155 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.409960    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97" podUID="971eae80-fd38-4164-8717-ade4cf3d5c8a" containerName="payload-processing" containerID="cri-o://c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9" gracePeriod=30
Jul 30 11:46:07.415221 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.415190    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"]
Jul 30 11:46:07.415406 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.415380    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9" podUID="f58bf83c-ac2c-4064-98ac-6a6761a68b3b" containerName="payload-pre-processing" containerID="cri-o://16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963" gracePeriod=30
Jul 30 11:46:07.458284 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.458252    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:07.560860 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.560823    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:07.666508 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.666447    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:46:07.673772 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.673743    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:07.692299 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.692278    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:46:07.782336 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.782303    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971eae80-fd38-4164-8717-ade4cf3d5c8a-plugins-config-volume\") pod \"971eae80-fd38-4164-8717-ade4cf3d5c8a\" (UID: \"971eae80-fd38-4164-8717-ade4cf3d5c8a\") "
Jul 30 11:46:07.782336 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.782335    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw9fd\" (UniqueName: \"kubernetes.io/projected/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-kube-api-access-fw9fd\") pod \"f58bf83c-ac2c-4064-98ac-6a6761a68b3b\" (UID: \"f58bf83c-ac2c-4064-98ac-6a6761a68b3b\") "
Jul 30 11:46:07.782542 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.782358    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-plugins-config-volume\") pod \"f58bf83c-ac2c-4064-98ac-6a6761a68b3b\" (UID: \"f58bf83c-ac2c-4064-98ac-6a6761a68b3b\") "
Jul 30 11:46:07.782542 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.782435    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xmqv\" (UniqueName: \"kubernetes.io/projected/971eae80-fd38-4164-8717-ade4cf3d5c8a-kube-api-access-7xmqv\") pod \"971eae80-fd38-4164-8717-ade4cf3d5c8a\" (UID: \"971eae80-fd38-4164-8717-ade4cf3d5c8a\") "
Jul 30 11:46:07.782840 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.782811    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f58bf83c-ac2c-4064-98ac-6a6761a68b3b" (UID: "f58bf83c-ac2c-4064-98ac-6a6761a68b3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:46:07.782840 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.782833    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971eae80-fd38-4164-8717-ade4cf3d5c8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "971eae80-fd38-4164-8717-ade4cf3d5c8a" (UID: "971eae80-fd38-4164-8717-ade4cf3d5c8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:46:07.784427 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.784408    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-kube-api-access-fw9fd" (OuterVolumeSpecName: "kube-api-access-fw9fd") pod "f58bf83c-ac2c-4064-98ac-6a6761a68b3b" (UID: "f58bf83c-ac2c-4064-98ac-6a6761a68b3b"). InnerVolumeSpecName "kube-api-access-fw9fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:46:07.784501 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.784465    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971eae80-fd38-4164-8717-ade4cf3d5c8a-kube-api-access-7xmqv" (OuterVolumeSpecName: "kube-api-access-7xmqv") pod "971eae80-fd38-4164-8717-ade4cf3d5c8a" (UID: "971eae80-fd38-4164-8717-ade4cf3d5c8a"). InnerVolumeSpecName "kube-api-access-7xmqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:46:07.883150 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.883118    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xmqv\" (UniqueName: \"kubernetes.io/projected/971eae80-fd38-4164-8717-ade4cf3d5c8a-kube-api-access-7xmqv\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:46:07.883150 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.883149    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971eae80-fd38-4164-8717-ade4cf3d5c8a-plugins-config-volume\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:46:07.883289 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.883162    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw9fd\" (UniqueName: \"kubernetes.io/projected/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-kube-api-access-fw9fd\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:46:07.883289 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.883176    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58bf83c-ac2c-4064-98ac-6a6761a68b3b-plugins-config-volume\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:46:07.963009 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:07.962976    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:08.457087 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.457051    2571 generic.go:356] "Generic (PLEG): container finished" podID="971eae80-fd38-4164-8717-ade4cf3d5c8a" containerID="c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9" exitCode=0
Jul 30 11:46:08.457481 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.457113    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"
Jul 30 11:46:08.457481 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.457140    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97" event={"ID":"971eae80-fd38-4164-8717-ade4cf3d5c8a","Type":"ContainerDied","Data":"c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9"}
Jul 30 11:46:08.457481 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.457198    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97" event={"ID":"971eae80-fd38-4164-8717-ade4cf3d5c8a","Type":"ContainerDied","Data":"f949270adf9799e88783f9efbbc8f702e07fb1d1831ac3ae0c97137a5504e8ff"}
Jul 30 11:46:08.457481 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.457227    2571 scope.go:117] "RemoveContainer" containerID="c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9"
Jul 30 11:46:08.458499 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.458479    2571 generic.go:356] "Generic (PLEG): container finished" podID="f58bf83c-ac2c-4064-98ac-6a6761a68b3b" containerID="16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963" exitCode=0
Jul 30 11:46:08.458562 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.458528    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9" event={"ID":"f58bf83c-ac2c-4064-98ac-6a6761a68b3b","Type":"ContainerDied","Data":"16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963"}
Jul 30 11:46:08.458603 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.458564    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9" event={"ID":"f58bf83c-ac2c-4064-98ac-6a6761a68b3b","Type":"ContainerDied","Data":"724b0e6feaa77879b6440256c4170f60de49f34816a9312b2c932e9de2063b44"}
Jul 30 11:46:08.458603 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.458532    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"
Jul 30 11:46:08.466750 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.466736    2571 scope.go:117] "RemoveContainer" containerID="c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9"
Jul 30 11:46:08.467139 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:46:08.466988    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9\": container with ID starting with c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9 not found: ID does not exist" containerID="c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9"
Jul 30 11:46:08.467139 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.467018    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9"} err="failed to get container status \"c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9\": rpc error: code = NotFound desc = could not find container \"c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9\": container with ID starting with c414522d11c1d668fb414d80af1d050bba7f8efec3a145d5c0f7d8a89baea7c9 not found: ID does not exist"
Jul 30 11:46:08.467139 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.467041    2571 scope.go:117] "RemoveContainer" containerID="16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963"
Jul 30 11:46:08.474615 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.474597    2571 scope.go:117] "RemoveContainer" containerID="16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963"
Jul 30 11:46:08.474872 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:46:08.474855    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963\": container with ID starting with 16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963 not found: ID does not exist" containerID="16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963"
Jul 30 11:46:08.474916 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.474881    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963"} err="failed to get container status \"16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963\": rpc error: code = NotFound desc = could not find container \"16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963\": container with ID starting with 16b6768f5f3572df7babe012f210e294299355dc7cc16454be4ed8ff932ec963 not found: ID does not exist"
Jul 30 11:46:08.483970 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.483947    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"]
Jul 30 11:46:08.485321 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.485292    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d1-5d96689ccb-57hw9"]
Jul 30 11:46:08.496457 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.496437    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"]
Jul 30 11:46:08.501965 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.501947    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d722d1-84bf95f866-wwj97"]
Jul 30 11:46:08.855440 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.855406    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="971eae80-fd38-4164-8717-ade4cf3d5c8a" path="/var/lib/kubelet/pods/971eae80-fd38-4164-8717-ade4cf3d5c8a/volumes"
Jul 30 11:46:08.855764 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:08.855751    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f58bf83c-ac2c-4064-98ac-6a6761a68b3b" path="/var/lib/kubelet/pods/f58bf83c-ac2c-4064-98ac-6a6761a68b3b/volumes"
Jul 30 11:46:09.680417 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:09.680387    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:46:12.935845 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:12.935813    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"]
Jul 30 11:46:13.035606 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:13.035568    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:27.954249 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:27.954209    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm" podUID="a79e5b28-863b-47fe-93b2-b027fc0adda2" containerName="main" containerID="cri-o://9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320" gracePeriod=45
Jul 30 11:46:28.205749 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.205683    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:46:28.360312 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360280    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-model-cache\") pod \"a79e5b28-863b-47fe-93b2-b027fc0adda2\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") "
Jul 30 11:46:28.360508 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360331    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7rv5\" (UniqueName: \"kubernetes.io/projected/a79e5b28-863b-47fe-93b2-b027fc0adda2-kube-api-access-c7rv5\") pod \"a79e5b28-863b-47fe-93b2-b027fc0adda2\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") "
Jul 30 11:46:28.360508 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360429    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-tmp-dir\") pod \"a79e5b28-863b-47fe-93b2-b027fc0adda2\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") "
Jul 30 11:46:28.360508 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360495    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79e5b28-863b-47fe-93b2-b027fc0adda2-tls-certs\") pod \"a79e5b28-863b-47fe-93b2-b027fc0adda2\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") "
Jul 30 11:46:28.360682 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360524    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-home\") pod \"a79e5b28-863b-47fe-93b2-b027fc0adda2\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") "
Jul 30 11:46:28.360682 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360551    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-dshm\") pod \"a79e5b28-863b-47fe-93b2-b027fc0adda2\" (UID: \"a79e5b28-863b-47fe-93b2-b027fc0adda2\") "
Jul 30 11:46:28.360682 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360609    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a79e5b28-863b-47fe-93b2-b027fc0adda2" (UID: "a79e5b28-863b-47fe-93b2-b027fc0adda2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:46:28.360831 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360673    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a79e5b28-863b-47fe-93b2-b027fc0adda2" (UID: "a79e5b28-863b-47fe-93b2-b027fc0adda2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:46:28.360831 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360792    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-home" (OuterVolumeSpecName: "home") pod "a79e5b28-863b-47fe-93b2-b027fc0adda2" (UID: "a79e5b28-863b-47fe-93b2-b027fc0adda2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:46:28.360917 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360830    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-model-cache\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:46:28.360917 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.360844    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-tmp-dir\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:46:28.362678 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.362648    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79e5b28-863b-47fe-93b2-b027fc0adda2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a79e5b28-863b-47fe-93b2-b027fc0adda2" (UID: "a79e5b28-863b-47fe-93b2-b027fc0adda2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 11:46:28.362785 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.362653    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-dshm" (OuterVolumeSpecName: "dshm") pod "a79e5b28-863b-47fe-93b2-b027fc0adda2" (UID: "a79e5b28-863b-47fe-93b2-b027fc0adda2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:46:28.362785 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.362686    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79e5b28-863b-47fe-93b2-b027fc0adda2-kube-api-access-c7rv5" (OuterVolumeSpecName: "kube-api-access-c7rv5") pod "a79e5b28-863b-47fe-93b2-b027fc0adda2" (UID: "a79e5b28-863b-47fe-93b2-b027fc0adda2"). InnerVolumeSpecName "kube-api-access-c7rv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:46:28.462267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.462185    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79e5b28-863b-47fe-93b2-b027fc0adda2-tls-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:46:28.462267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.462223    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-home\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:46:28.462267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.462233    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79e5b28-863b-47fe-93b2-b027fc0adda2-dshm\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:46:28.462267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.462241    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7rv5\" (UniqueName: \"kubernetes.io/projected/a79e5b28-863b-47fe-93b2-b027fc0adda2-kube-api-access-c7rv5\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:46:28.543328 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.543298    2571 generic.go:356] "Generic (PLEG): container finished" podID="a79e5b28-863b-47fe-93b2-b027fc0adda2" containerID="9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320" exitCode=0
Jul 30 11:46:28.543493 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.543343    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"
Jul 30 11:46:28.543493 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.543370    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm" event={"ID":"a79e5b28-863b-47fe-93b2-b027fc0adda2","Type":"ContainerDied","Data":"9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320"}
Jul 30 11:46:28.543493 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.543399    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm" event={"ID":"a79e5b28-863b-47fe-93b2-b027fc0adda2","Type":"ContainerDied","Data":"9765d41894ad9159a73a9400cd4cc2e00f4d2adcbfd18ef3ed86d08f5037d898"}
Jul 30 11:46:28.543493 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.543417    2571 scope.go:117] "RemoveContainer" containerID="9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320"
Jul 30 11:46:28.558636 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.558617    2571 scope.go:117] "RemoveContainer" containerID="9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320"
Jul 30 11:46:28.558937 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:46:28.558916    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320\": container with ID starting with 9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320 not found: ID does not exist" containerID="9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320"
Jul 30 11:46:28.559015 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.558943    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320"} err="failed to get container status \"9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320\": rpc error: code = NotFound desc = could not find container \"9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320\": container with ID starting with 9a831d87d35bdbf01338defab7c5b9d8d0871323b22f72d6169dbee9ea858320 not found: ID does not exist"
Jul 30 11:46:28.568497 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.568474    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"]
Jul 30 11:46:28.574252 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.574234    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d722d1/ipp-route-d722d1-kserve-56b44b8d55-mvpvm"]
Jul 30 11:46:28.854614 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:28.854533    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a79e5b28-863b-47fe-93b2-b027fc0adda2" path="/var/lib/kubelet/pods/a79e5b28-863b-47fe-93b2-b027fc0adda2/volumes"
Jul 30 11:46:49.862500 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:49.862414    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:51.696683 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:51.696648    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:58.697547 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:58.697507    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:58.808186 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:58.808149    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:58.954204 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:58.954120    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"]
Jul 30 11:46:58.954374 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:58.954352    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy" containerID="cri-o://8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2" gracePeriod=30
Jul 30 11:46:58.964691 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:58.964651    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:59.232877 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.232795    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:46:59.413857 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.413823    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"]
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414628    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f58bf83c-ac2c-4064-98ac-6a6761a68b3b" containerName="payload-pre-processing"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414660    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f58bf83c-ac2c-4064-98ac-6a6761a68b3b" containerName="payload-pre-processing"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414702    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79e5b28-863b-47fe-93b2-b027fc0adda2" containerName="main"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414712    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79e5b28-863b-47fe-93b2-b027fc0adda2" containerName="main"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414732    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414741    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414753    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414762    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414789    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971eae80-fd38-4164-8717-ade4cf3d5c8a" containerName="payload-processing"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414798    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="971eae80-fd38-4164-8717-ade4cf3d5c8a" containerName="payload-processing"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.414998    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f58bf83c-ac2c-4064-98ac-6a6761a68b3b" containerName="payload-pre-processing"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.415011    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.415020    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="971eae80-fd38-4164-8717-ade4cf3d5c8a" containerName="payload-processing"
Jul 30 11:46:59.416506 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.415035    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79e5b28-863b-47fe-93b2-b027fc0adda2" containerName="main"
Jul 30 11:46:59.418133 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.418091    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"
Jul 30 11:46:59.422589 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.422567    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\""
Jul 30 11:46:59.433042 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.433022    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 11:46:59.438042 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.438020    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"]
Jul 30 11:46:59.500127 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.500051    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d55c2a-088a-4549-b5e4-700cdbe0e0fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4\" (UID: \"26d55c2a-088a-4549-b5e4-700cdbe0e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"
Jul 30 11:46:59.600704 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.600675    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d55c2a-088a-4549-b5e4-700cdbe0e0fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4\" (UID: \"26d55c2a-088a-4549-b5e4-700cdbe0e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"
Jul 30 11:46:59.601323 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.601303    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d55c2a-088a-4549-b5e4-700cdbe0e0fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4\" (UID: \"26d55c2a-088a-4549-b5e4-700cdbe0e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"
Jul 30 11:46:59.728977 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.728952    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"
Jul 30 11:46:59.875382 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:46:59.875350    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"]
Jul 30 11:46:59.877930 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:46:59.877900    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d55c2a_088a_4549_b5e4_700cdbe0e0fe.slice/crio-3fbbe1e8b437698a3858bea6307de4b2b68cf7eadc64a7dd65eadee36e21527c WatchSource:0}: Error finding container 3fbbe1e8b437698a3858bea6307de4b2b68cf7eadc64a7dd65eadee36e21527c: Status 404 returned error can't find the container with id 3fbbe1e8b437698a3858bea6307de4b2b68cf7eadc64a7dd65eadee36e21527c
Jul 30 11:47:00.677425 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:00.677388    2571 generic.go:356] "Generic (PLEG): container finished" podID="26d55c2a-088a-4549-b5e4-700cdbe0e0fe" containerID="531a91a16a22f940f2876a8f57dd1c4080e8621a19d1359f66499452e8bef1dc" exitCode=0
Jul 30 11:47:00.677609 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:00.677475    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4" event={"ID":"26d55c2a-088a-4549-b5e4-700cdbe0e0fe","Type":"ContainerDied","Data":"531a91a16a22f940f2876a8f57dd1c4080e8621a19d1359f66499452e8bef1dc"}
Jul 30 11:47:00.677609 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:00.677540    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4" event={"ID":"26d55c2a-088a-4549-b5e4-700cdbe0e0fe","Type":"ContainerStarted","Data":"3fbbe1e8b437698a3858bea6307de4b2b68cf7eadc64a7dd65eadee36e21527c"}
Jul 30 11:47:01.810305 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:01.810279    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"
Jul 30 11:47:01.814227 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:01.814205    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d55c2a-088a-4549-b5e4-700cdbe0e0fe-maas-api-service-ca\") pod \"26d55c2a-088a-4549-b5e4-700cdbe0e0fe\" (UID: \"26d55c2a-088a-4549-b5e4-700cdbe0e0fe\") "
Jul 30 11:47:01.814514 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:01.814493    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d55c2a-088a-4549-b5e4-700cdbe0e0fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26d55c2a-088a-4549-b5e4-700cdbe0e0fe" (UID: "26d55c2a-088a-4549-b5e4-700cdbe0e0fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:47:01.914845 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:01.914811    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d55c2a-088a-4549-b5e4-700cdbe0e0fe-maas-api-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:02.686806 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:02.686772    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4" event={"ID":"26d55c2a-088a-4549-b5e4-700cdbe0e0fe","Type":"ContainerDied","Data":"3fbbe1e8b437698a3858bea6307de4b2b68cf7eadc64a7dd65eadee36e21527c"}
Jul 30 11:47:02.686985 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:02.686806    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"
Jul 30 11:47:02.686985 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:02.686821    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fbbe1e8b437698a3858bea6307de4b2b68cf7eadc64a7dd65eadee36e21527c"
Jul 30 11:47:04.197673 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.197645    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:47:04.231710 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.231636    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4krg\" (UniqueName: \"kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-kube-api-access-j4krg\") pod \"8f2b5666-c22c-4672-971e-842348f476c6\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") "
Jul 30 11:47:04.231710 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.231678    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2b5666-c22c-4672-971e-842348f476c6-istio-podinfo\") pod \"8f2b5666-c22c-4672-971e-842348f476c6\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") "
Jul 30 11:47:04.231710 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.231709    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-certs\") pod \"8f2b5666-c22c-4672-971e-842348f476c6\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") "
Jul 30 11:47:04.231966 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.231748    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-socket\") pod \"8f2b5666-c22c-4672-971e-842348f476c6\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") "
Jul 30 11:47:04.231966 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.231774    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-credential-socket\") pod \"8f2b5666-c22c-4672-971e-842348f476c6\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") "
Jul 30 11:47:04.231966 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.231911    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-data\") pod \"8f2b5666-c22c-4672-971e-842348f476c6\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") "
Jul 30 11:47:04.232195 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232014    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2b5666-c22c-4672-971e-842348f476c6-istiod-ca-cert\") pod \"8f2b5666-c22c-4672-971e-842348f476c6\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") "
Jul 30 11:47:04.232195 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232041    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-istio-token\") pod \"8f2b5666-c22c-4672-971e-842348f476c6\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") "
Jul 30 11:47:04.232195 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232050    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f2b5666-c22c-4672-971e-842348f476c6" (UID: "8f2b5666-c22c-4672-971e-842348f476c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:47:04.232195 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232074    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-envoy\") pod \"8f2b5666-c22c-4672-971e-842348f476c6\" (UID: \"8f2b5666-c22c-4672-971e-842348f476c6\") "
Jul 30 11:47:04.232195 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232078    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f2b5666-c22c-4672-971e-842348f476c6" (UID: "8f2b5666-c22c-4672-971e-842348f476c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:47:04.232195 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232121    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f2b5666-c22c-4672-971e-842348f476c6" (UID: "8f2b5666-c22c-4672-971e-842348f476c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:47:04.232507 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232248    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f2b5666-c22c-4672-971e-842348f476c6" (UID: "8f2b5666-c22c-4672-971e-842348f476c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:47:04.232507 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232329    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2b5666-c22c-4672-971e-842348f476c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f2b5666-c22c-4672-971e-842348f476c6" (UID: "8f2b5666-c22c-4672-971e-842348f476c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:47:04.232507 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232352    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-socket\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:04.232507 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232369    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-credential-socket\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:04.232507 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232379    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-data\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:04.232507 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.232391    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-workload-certs\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:04.234053 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.234031    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f2b5666-c22c-4672-971e-842348f476c6" (UID: "8f2b5666-c22c-4672-971e-842348f476c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:47:04.234424 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.234392    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-kube-api-access-j4krg" (OuterVolumeSpecName: "kube-api-access-j4krg") pod "8f2b5666-c22c-4672-971e-842348f476c6" (UID: "8f2b5666-c22c-4672-971e-842348f476c6"). InnerVolumeSpecName "kube-api-access-j4krg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:47:04.234530 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.234453    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f2b5666-c22c-4672-971e-842348f476c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f2b5666-c22c-4672-971e-842348f476c6" (UID: "8f2b5666-c22c-4672-971e-842348f476c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 11:47:04.234530 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.234500    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f2b5666-c22c-4672-971e-842348f476c6" (UID: "8f2b5666-c22c-4672-971e-842348f476c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 11:47:04.332995 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.332957    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2b5666-c22c-4672-971e-842348f476c6-istiod-ca-cert\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:04.332995 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.332989    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-istio-token\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:04.332995 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.332997    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2b5666-c22c-4672-971e-842348f476c6-istio-envoy\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:04.332995 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.333006    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4krg\" (UniqueName: \"kubernetes.io/projected/8f2b5666-c22c-4672-971e-842348f476c6-kube-api-access-j4krg\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:04.333325 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.333016    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2b5666-c22c-4672-971e-842348f476c6-istio-podinfo\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:04.696009 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.695975    2571 generic.go:356] "Generic (PLEG): container finished" podID="8f2b5666-c22c-4672-971e-842348f476c6" containerID="8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2" exitCode=0
Jul 30 11:47:04.696210 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.696023    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"
Jul 30 11:47:04.696210 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.696046    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" event={"ID":"8f2b5666-c22c-4672-971e-842348f476c6","Type":"ContainerDied","Data":"8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2"}
Jul 30 11:47:04.696210 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.696117    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7" event={"ID":"8f2b5666-c22c-4672-971e-842348f476c6","Type":"ContainerDied","Data":"ad5e8fae018ec9e08ccd85a0408a2b88b3ce38a5846d827a286243249ecd2d82"}
Jul 30 11:47:04.696210 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.696149    2571 scope.go:117] "RemoveContainer" containerID="8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2"
Jul 30 11:47:04.705815 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.705796    2571 scope.go:117] "RemoveContainer" containerID="8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b"
Jul 30 11:47:04.715165 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.715001    2571 scope.go:117] "RemoveContainer" containerID="8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2"
Jul 30 11:47:04.716437 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:47:04.716406    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2\": container with ID starting with 8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2 not found: ID does not exist" containerID="8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2"
Jul 30 11:47:04.716527 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.716445    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2"} err="failed to get container status \"8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2\": rpc error: code = NotFound desc = could not find container \"8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2\": container with ID starting with 8f1c9e152a7172bb8bbaee27a2e36b92e2a87127697d6bab7dc45dd695694de2 not found: ID does not exist"
Jul 30 11:47:04.716527 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.716463    2571 scope.go:117] "RemoveContainer" containerID="8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b"
Jul 30 11:47:04.716706 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:47:04.716687    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b\": container with ID starting with 8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b not found: ID does not exist" containerID="8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b"
Jul 30 11:47:04.716761 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.716712    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b"} err="failed to get container status \"8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b\": rpc error: code = NotFound desc = could not find container \"8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b\": container with ID starting with 8fc741b55bbd5f926a6e8db052472baf16a06f4c4f740e5d1f7dae301d4e9e0b not found: ID does not exist"
Jul 30 11:47:04.734247 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.734223    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"]
Jul 30 11:47:04.735386 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.735365    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d722d1-openshift-default-9486b5cd6-fxhp7"]
Jul 30 11:47:04.854394 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:04.854362    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f2b5666-c22c-4672-971e-842348f476c6" path="/var/lib/kubelet/pods/8f2b5666-c22c-4672-971e-842348f476c6/volumes"
Jul 30 11:47:05.534773 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.534720    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:05.552570 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.552540    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"]
Jul 30 11:47:05.553008 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.552979    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8" podUID="13f67b57-4dd3-4608-b068-3300925414ee" containerName="payload-processing" containerID="cri-o://0940f006dc34ed19de16842ffb0960fbc677416fb6632fa38560362de396dbda" gracePeriod=30
Jul 30 11:47:05.561845 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.561818    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"]
Jul 30 11:47:05.562093 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.562063    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w" podUID="d074effb-916b-4917-8a25-7d59ec3d12c5" containerName="payload-pre-processing" containerID="cri-o://43eb7f0f2d4adf25a9bcc34e1f739df2f99bb9ee9321f70e8b115dc2434dfab1" gracePeriod=30
Jul 30 11:47:05.612138 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.612069    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:05.697125 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.696170    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"]
Jul 30 11:47:05.699638 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.699524    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:05.699790 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.699663    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0ce38-9gpk4"]
Jul 30 11:47:05.705045 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.705018    2571 generic.go:356] "Generic (PLEG): container finished" podID="d074effb-916b-4917-8a25-7d59ec3d12c5" containerID="43eb7f0f2d4adf25a9bcc34e1f739df2f99bb9ee9321f70e8b115dc2434dfab1" exitCode=0
Jul 30 11:47:05.705198 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.705053    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w" event={"ID":"d074effb-916b-4917-8a25-7d59ec3d12c5","Type":"ContainerDied","Data":"43eb7f0f2d4adf25a9bcc34e1f739df2f99bb9ee9321f70e8b115dc2434dfab1"}
Jul 30 11:47:05.706955 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.706861    2571 generic.go:356] "Generic (PLEG): container finished" podID="13f67b57-4dd3-4608-b068-3300925414ee" containerID="0940f006dc34ed19de16842ffb0960fbc677416fb6632fa38560362de396dbda" exitCode=0
Jul 30 11:47:05.706955 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.706935    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8" event={"ID":"13f67b57-4dd3-4608-b068-3300925414ee","Type":"ContainerDied","Data":"0940f006dc34ed19de16842ffb0960fbc677416fb6632fa38560362de396dbda"}
Jul 30 11:47:05.775462 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.775403    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:05.826045 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.826018    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:47:05.829223 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.829193    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:47:05.843167 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.843146    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbhsn\" (UniqueName: \"kubernetes.io/projected/d074effb-916b-4917-8a25-7d59ec3d12c5-kube-api-access-sbhsn\") pod \"d074effb-916b-4917-8a25-7d59ec3d12c5\" (UID: \"d074effb-916b-4917-8a25-7d59ec3d12c5\") "
Jul 30 11:47:05.843267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.843184    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8p7g\" (UniqueName: \"kubernetes.io/projected/13f67b57-4dd3-4608-b068-3300925414ee-kube-api-access-z8p7g\") pod \"13f67b57-4dd3-4608-b068-3300925414ee\" (UID: \"13f67b57-4dd3-4608-b068-3300925414ee\") "
Jul 30 11:47:05.843267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.843205    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f67b57-4dd3-4608-b068-3300925414ee-plugins-config-volume\") pod \"13f67b57-4dd3-4608-b068-3300925414ee\" (UID: \"13f67b57-4dd3-4608-b068-3300925414ee\") "
Jul 30 11:47:05.843267 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.843251    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d074effb-916b-4917-8a25-7d59ec3d12c5-plugins-config-volume\") pod \"d074effb-916b-4917-8a25-7d59ec3d12c5\" (UID: \"d074effb-916b-4917-8a25-7d59ec3d12c5\") "
Jul 30 11:47:05.843584 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.843562    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f67b57-4dd3-4608-b068-3300925414ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13f67b57-4dd3-4608-b068-3300925414ee" (UID: "13f67b57-4dd3-4608-b068-3300925414ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:47:05.843636 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.843595    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d074effb-916b-4917-8a25-7d59ec3d12c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d074effb-916b-4917-8a25-7d59ec3d12c5" (UID: "d074effb-916b-4917-8a25-7d59ec3d12c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:47:05.845472 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.845451    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f67b57-4dd3-4608-b068-3300925414ee-kube-api-access-z8p7g" (OuterVolumeSpecName: "kube-api-access-z8p7g") pod "13f67b57-4dd3-4608-b068-3300925414ee" (UID: "13f67b57-4dd3-4608-b068-3300925414ee"). InnerVolumeSpecName "kube-api-access-z8p7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:47:05.845569 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.845507    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d074effb-916b-4917-8a25-7d59ec3d12c5-kube-api-access-sbhsn" (OuterVolumeSpecName: "kube-api-access-sbhsn") pod "d074effb-916b-4917-8a25-7d59ec3d12c5" (UID: "d074effb-916b-4917-8a25-7d59ec3d12c5"). InnerVolumeSpecName "kube-api-access-sbhsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:47:05.943717 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.943687    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbhsn\" (UniqueName: \"kubernetes.io/projected/d074effb-916b-4917-8a25-7d59ec3d12c5-kube-api-access-sbhsn\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:05.943717 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.943711    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8p7g\" (UniqueName: \"kubernetes.io/projected/13f67b57-4dd3-4608-b068-3300925414ee-kube-api-access-z8p7g\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:05.943717 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.943722    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f67b57-4dd3-4608-b068-3300925414ee-plugins-config-volume\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:05.943952 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:05.943731    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d074effb-916b-4917-8a25-7d59ec3d12c5-plugins-config-volume\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:06.099010 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.098909    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:06.712180 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.712148    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"
Jul 30 11:47:06.712180 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.712167    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w" event={"ID":"d074effb-916b-4917-8a25-7d59ec3d12c5","Type":"ContainerDied","Data":"c7fee43c7c09c068fb80992942d2f14a5201bfe3bf17d25563d05175bc4564bf"}
Jul 30 11:47:06.712687 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.712241    2571 scope.go:117] "RemoveContainer" containerID="43eb7f0f2d4adf25a9bcc34e1f739df2f99bb9ee9321f70e8b115dc2434dfab1"
Jul 30 11:47:06.713676 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.713654    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8" event={"ID":"13f67b57-4dd3-4608-b068-3300925414ee","Type":"ContainerDied","Data":"d751b09def52ff0cddb2816cdd008f271c9c922b2597bcca56cd488a9a421754"}
Jul 30 11:47:06.713676 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.713670    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"
Jul 30 11:47:06.721555 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.721537    2571 scope.go:117] "RemoveContainer" containerID="0940f006dc34ed19de16842ffb0960fbc677416fb6632fa38560362de396dbda"
Jul 30 11:47:06.734824 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.734803    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"]
Jul 30 11:47:06.739261 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.739241    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0ce38-cd6664556-cpq7w"]
Jul 30 11:47:06.752121 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.752088    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"]
Jul 30 11:47:06.758717 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.758698    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0ce38-57d7c54694-s7sl8"]
Jul 30 11:47:06.855585 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.855508    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f67b57-4dd3-4608-b068-3300925414ee" path="/var/lib/kubelet/pods/13f67b57-4dd3-4608-b068-3300925414ee/volumes"
Jul 30 11:47:06.855825 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.855812    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26d55c2a-088a-4549-b5e4-700cdbe0e0fe" path="/var/lib/kubelet/pods/26d55c2a-088a-4549-b5e4-700cdbe0e0fe/volumes"
Jul 30 11:47:06.856155 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:06.856142    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d074effb-916b-4917-8a25-7d59ec3d12c5" path="/var/lib/kubelet/pods/d074effb-916b-4917-8a25-7d59ec3d12c5/volumes"
Jul 30 11:47:07.894896 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:07.894862    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:47:12.578462 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:12.578425    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:12.636325 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:12.636293    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:12.845526 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:12.845439    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:13.094380 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.094349    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:13.243238 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243199    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"]
Jul 30 11:47:13.243545 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243528    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d55c2a-088a-4549-b5e4-700cdbe0e0fe" containerName="revoke-keys"
Jul 30 11:47:13.243628 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243548    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d55c2a-088a-4549-b5e4-700cdbe0e0fe" containerName="revoke-keys"
Jul 30 11:47:13.243628 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243562    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d074effb-916b-4917-8a25-7d59ec3d12c5" containerName="payload-pre-processing"
Jul 30 11:47:13.243628 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243569    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d074effb-916b-4917-8a25-7d59ec3d12c5" containerName="payload-pre-processing"
Jul 30 11:47:13.243628 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243579    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f67b57-4dd3-4608-b068-3300925414ee" containerName="payload-processing"
Jul 30 11:47:13.243628 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243588    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f67b57-4dd3-4608-b068-3300925414ee" containerName="payload-processing"
Jul 30 11:47:13.243628 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243602    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy"
Jul 30 11:47:13.243628 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243610    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy"
Jul 30 11:47:13.243974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243639    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy"
Jul 30 11:47:13.243974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243649    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy"
Jul 30 11:47:13.243974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243721    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy"
Jul 30 11:47:13.243974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243732    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d074effb-916b-4917-8a25-7d59ec3d12c5" containerName="payload-pre-processing"
Jul 30 11:47:13.243974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243746    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f67b57-4dd3-4608-b068-3300925414ee" containerName="payload-processing"
Jul 30 11:47:13.243974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243756    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d55c2a-088a-4549-b5e4-700cdbe0e0fe" containerName="revoke-keys"
Jul 30 11:47:13.243974 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.243764    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6039dbd6-58d1-4ab9-8f98-9940b7c56406" containerName="istio-proxy"
Jul 30 11:47:13.246394 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.246373    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"
Jul 30 11:47:13.249158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.249136    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 11:47:13.249277 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.249261    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\""
Jul 30 11:47:13.254853 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.254826    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"]
Jul 30 11:47:13.301824 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.301792    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5476a76-0040-444f-a398-bc01750d6f11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9\" (UID: \"c5476a76-0040-444f-a398-bc01750d6f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"
Jul 30 11:47:13.402368 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.402340    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5476a76-0040-444f-a398-bc01750d6f11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9\" (UID: \"c5476a76-0040-444f-a398-bc01750d6f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"
Jul 30 11:47:13.402903 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.402883    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5476a76-0040-444f-a398-bc01750d6f11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9\" (UID: \"c5476a76-0040-444f-a398-bc01750d6f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"
Jul 30 11:47:13.556761 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.556677    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"
Jul 30 11:47:13.679977 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.679949    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"]
Jul 30 11:47:13.682735 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:47:13.682710    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5476a76_0040_444f_a398_bc01750d6f11.slice/crio-fda74b83f840f7c64b4b569ef9b75de2819618326825dce2bef74b5768521f85 WatchSource:0}: Error finding container fda74b83f840f7c64b4b569ef9b75de2819618326825dce2bef74b5768521f85: Status 404 returned error can't find the container with id fda74b83f840f7c64b4b569ef9b75de2819618326825dce2bef74b5768521f85
Jul 30 11:47:13.747398 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:13.747374    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9" event={"ID":"c5476a76-0040-444f-a398-bc01750d6f11","Type":"ContainerStarted","Data":"fda74b83f840f7c64b4b569ef9b75de2819618326825dce2bef74b5768521f85"}
Jul 30 11:47:14.752888 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:14.752851    2571 generic.go:356] "Generic (PLEG): container finished" podID="c5476a76-0040-444f-a398-bc01750d6f11" containerID="fdb91bc9277423836a0870e5693fe58edaa8dac85cd70264412ddbcd293430a5" exitCode=0
Jul 30 11:47:14.753314 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:14.752937    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9" event={"ID":"c5476a76-0040-444f-a398-bc01750d6f11","Type":"ContainerDied","Data":"fdb91bc9277423836a0870e5693fe58edaa8dac85cd70264412ddbcd293430a5"}
Jul 30 11:47:15.878050 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:15.878023    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"
Jul 30 11:47:15.922992 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:15.922966    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5476a76-0040-444f-a398-bc01750d6f11-maas-api-service-ca\") pod \"c5476a76-0040-444f-a398-bc01750d6f11\" (UID: \"c5476a76-0040-444f-a398-bc01750d6f11\") "
Jul 30 11:47:15.923596 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:15.923358    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5476a76-0040-444f-a398-bc01750d6f11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5476a76-0040-444f-a398-bc01750d6f11" (UID: "c5476a76-0040-444f-a398-bc01750d6f11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:47:16.024560 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:16.024486    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5476a76-0040-444f-a398-bc01750d6f11-maas-api-service-ca\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:16.765427 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:16.765395    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"
Jul 30 11:47:16.765609 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:16.765439    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9" event={"ID":"c5476a76-0040-444f-a398-bc01750d6f11","Type":"ContainerDied","Data":"fda74b83f840f7c64b4b569ef9b75de2819618326825dce2bef74b5768521f85"}
Jul 30 11:47:16.765609 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:16.765490    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fda74b83f840f7c64b4b569ef9b75de2819618326825dce2bef74b5768521f85"
Jul 30 11:47:20.303986 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:20.303951    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:20.371087 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:20.371036    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:20.457663 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:20.457627    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:20.610993 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:20.610912    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:20.708939 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:20.708908    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"]
Jul 30 11:47:20.711522 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:20.711497    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-060e9f-gmjg9"]
Jul 30 11:47:20.854908 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:20.854872    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5476a76-0040-444f-a398-bc01750d6f11" path="/var/lib/kubelet/pods/c5476a76-0040-444f-a398-bc01750d6f11/volumes"
Jul 30 11:47:21.075661 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:21.075627    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:22.590645 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:22.590605    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:47:27.993545 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:27.993506    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:28.099373 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:28.099328    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:28.257517 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:28.257421    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:28.495859 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:28.495806    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:35.124592 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.124546    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:35.166920 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.166883    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"]
Jul 30 11:47:35.167220 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.167194    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz" podUID="0af58b7e-d20d-47bf-a07c-b9c43ca312f2" containerName="payload-processing" containerID="cri-o://d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d" gracePeriod=30
Jul 30 11:47:35.210053 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.210019    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:35.292244 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.292204    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:35.375033 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.374936    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:35.417086 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.417059    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:47:35.471969 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.471937    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4cvw\" (UniqueName: \"kubernetes.io/projected/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-kube-api-access-b4cvw\") pod \"0af58b7e-d20d-47bf-a07c-b9c43ca312f2\" (UID: \"0af58b7e-d20d-47bf-a07c-b9c43ca312f2\") "
Jul 30 11:47:35.472131 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.472035    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-plugins-config-volume\") pod \"0af58b7e-d20d-47bf-a07c-b9c43ca312f2\" (UID: \"0af58b7e-d20d-47bf-a07c-b9c43ca312f2\") "
Jul 30 11:47:35.472414 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.472391    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0af58b7e-d20d-47bf-a07c-b9c43ca312f2" (UID: "0af58b7e-d20d-47bf-a07c-b9c43ca312f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 11:47:35.474046 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.474026    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-kube-api-access-b4cvw" (OuterVolumeSpecName: "kube-api-access-b4cvw") pod "0af58b7e-d20d-47bf-a07c-b9c43ca312f2" (UID: "0af58b7e-d20d-47bf-a07c-b9c43ca312f2"). InnerVolumeSpecName "kube-api-access-b4cvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 11:47:35.573279 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.573247    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4cvw\" (UniqueName: \"kubernetes.io/projected/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-kube-api-access-b4cvw\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:35.573279 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.573271    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af58b7e-d20d-47bf-a07c-b9c43ca312f2-plugins-config-volume\") on node \"ip-10-0-128-243.ec2.internal\" DevicePath \"\""
Jul 30 11:47:35.846701 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.846668    2571 generic.go:356] "Generic (PLEG): container finished" podID="0af58b7e-d20d-47bf-a07c-b9c43ca312f2" containerID="d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d" exitCode=0
Jul 30 11:47:35.846863 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.846710    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"
Jul 30 11:47:35.846863 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.846747    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz" event={"ID":"0af58b7e-d20d-47bf-a07c-b9c43ca312f2","Type":"ContainerDied","Data":"d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d"}
Jul 30 11:47:35.846863 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.846799    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz" event={"ID":"0af58b7e-d20d-47bf-a07c-b9c43ca312f2","Type":"ContainerDied","Data":"e5516faf4c07c53ec0ab93c8ff6564b6e968a0046515eb5c971aa3b050432efa"}
Jul 30 11:47:35.846863 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.846822    2571 scope.go:117] "RemoveContainer" containerID="d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d"
Jul 30 11:47:35.855762 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.855745    2571 scope.go:117] "RemoveContainer" containerID="d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d"
Jul 30 11:47:35.856030 ip-10-0-128-243 kubenswrapper[2571]: E0730 11:47:35.856009    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d\": container with ID starting with d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d not found: ID does not exist" containerID="d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d"
Jul 30 11:47:35.856129 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.856037    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d"} err="failed to get container status \"d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d\": rpc error: code = NotFound desc = could not find container \"d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d\": container with ID starting with d864285b83759b8fddf1555a93a28dd98c9cba990625de2480cdd635bdd5122d not found: ID does not exist"
Jul 30 11:47:35.868753 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.868730    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"]
Jul 30 11:47:35.874593 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.874571    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-945db9-56755b4f9d-sxnhz"]
Jul 30 11:47:35.890278 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:35.890227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:36.853705 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:36.853667    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af58b7e-d20d-47bf-a07c-b9c43ca312f2" path="/var/lib/kubelet/pods/0af58b7e-d20d-47bf-a07c-b9c43ca312f2/volumes"
Jul 30 11:47:37.204225 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:37.204192    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-sthk2"]
Jul 30 11:47:42.021967 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:42.021925    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:42.084290 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:42.084252    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:42.291380 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:42.291301    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:42.515057 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:42.515014    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:42.943203 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:42.943171    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f6748df59-rb6zl_fe25eefe-9d08-4719-a944-7859dcf49aff/authorino/0.log"
Jul 30 11:47:49.128092 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:49.128063    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d97895fdd-4dqwm_cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba/copy-manifests/0.log"
Jul 30 11:47:49.151973 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:49.151941    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d97895fdd-4dqwm_cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba/manager/0.log"
Jul 30 11:47:49.973592 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:49.973558    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756850-29k4f_91f363fa-8174-4186-99f1-51ef10897688/cleanup/0.log"
Jul 30 11:47:51.022084 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:51.022053    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f6748df59-rb6zl_fe25eefe-9d08-4719-a944-7859dcf49aff/authorino/0.log"
Jul 30 11:47:51.366471 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:51.366390    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-j6kf2_1f1ee5b9-b701-4c17-8eb0-6b1c55b22672/kuadrant-console-plugin/0.log"
Jul 30 11:47:51.596260 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:51.596221    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-sthk2_fc0d75c4-8753-48f0-8104-9bc0e2304a55/limitador/0.log"
Jul 30 11:47:52.317058 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:52.317028    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8674569cb6-mc94c_de38d89b-a26e-4d4d-999f-00a1585e9484/kube-auth-proxy/0.log"
Jul 30 11:47:53.955378 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:53.955345    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xf6d2_83d4ceca-d270-404e-a301-a0a840cb5ef7/main/0.log"
Jul 30 11:47:59.098749 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.098717    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6ms7b/must-gather-958pc"]
Jul 30 11:47:59.099158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.099004    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af58b7e-d20d-47bf-a07c-b9c43ca312f2" containerName="payload-processing"
Jul 30 11:47:59.099158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.099015    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af58b7e-d20d-47bf-a07c-b9c43ca312f2" containerName="payload-processing"
Jul 30 11:47:59.099158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.099024    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5476a76-0040-444f-a398-bc01750d6f11" containerName="revoke-keys"
Jul 30 11:47:59.099158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.099030    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5476a76-0040-444f-a398-bc01750d6f11" containerName="revoke-keys"
Jul 30 11:47:59.099158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.099082    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5476a76-0040-444f-a398-bc01750d6f11" containerName="revoke-keys"
Jul 30 11:47:59.099158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.099089    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2b5666-c22c-4672-971e-842348f476c6" containerName="istio-proxy"
Jul 30 11:47:59.099158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.099109    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af58b7e-d20d-47bf-a07c-b9c43ca312f2" containerName="payload-processing"
Jul 30 11:47:59.101972 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.101945    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ms7b/must-gather-958pc"
Jul 30 11:47:59.104066 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.104035    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:59.106453 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.106433    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6ms7b\"/\"openshift-service-ca.crt\""
Jul 30 11:47:59.106565 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.106494    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6ms7b\"/\"kube-root-ca.crt\""
Jul 30 11:47:59.107733 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.107714    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6ms7b\"/\"default-dockercfg-4xx59\""
Jul 30 11:47:59.119158 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.119134    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ms7b/must-gather-958pc"]
Jul 30 11:47:59.191670 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.191638    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:47:59.264723 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.264687    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1ee3895e-e740-40d8-aa6e-770b66f759b4-must-gather-output\") pod \"must-gather-958pc\" (UID: \"1ee3895e-e740-40d8-aa6e-770b66f759b4\") " pod="openshift-must-gather-6ms7b/must-gather-958pc"
Jul 30 11:47:59.264873 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.264741    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw4vp\" (UniqueName: \"kubernetes.io/projected/1ee3895e-e740-40d8-aa6e-770b66f759b4-kube-api-access-hw4vp\") pod \"must-gather-958pc\" (UID: \"1ee3895e-e740-40d8-aa6e-770b66f759b4\") " pod="openshift-must-gather-6ms7b/must-gather-958pc"
Jul 30 11:47:59.365546 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.365444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1ee3895e-e740-40d8-aa6e-770b66f759b4-must-gather-output\") pod \"must-gather-958pc\" (UID: \"1ee3895e-e740-40d8-aa6e-770b66f759b4\") " pod="openshift-must-gather-6ms7b/must-gather-958pc"
Jul 30 11:47:59.365546 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.365527    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4vp\" (UniqueName: \"kubernetes.io/projected/1ee3895e-e740-40d8-aa6e-770b66f759b4-kube-api-access-hw4vp\") pod \"must-gather-958pc\" (UID: \"1ee3895e-e740-40d8-aa6e-770b66f759b4\") " pod="openshift-must-gather-6ms7b/must-gather-958pc"
Jul 30 11:47:59.365879 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.365855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1ee3895e-e740-40d8-aa6e-770b66f759b4-must-gather-output\") pod \"must-gather-958pc\" (UID: \"1ee3895e-e740-40d8-aa6e-770b66f759b4\") " pod="openshift-must-gather-6ms7b/must-gather-958pc"
Jul 30 11:47:59.375397 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.375373    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4vp\" (UniqueName: \"kubernetes.io/projected/1ee3895e-e740-40d8-aa6e-770b66f759b4-kube-api-access-hw4vp\") pod \"must-gather-958pc\" (UID: \"1ee3895e-e740-40d8-aa6e-770b66f759b4\") " pod="openshift-must-gather-6ms7b/must-gather-958pc"
Jul 30 11:47:59.411415 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.411388    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ms7b/must-gather-958pc"
Jul 30 11:47:59.537939 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.537908    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ms7b/must-gather-958pc"]
Jul 30 11:47:59.540057 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:47:59.540028    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ee3895e_e740_40d8_aa6e_770b66f759b4.slice/crio-79c8b2c2aa889a93498110b90fb71c8863ae1defe89791b08610e2528d1cc6d6 WatchSource:0}: Error finding container 79c8b2c2aa889a93498110b90fb71c8863ae1defe89791b08610e2528d1cc6d6: Status 404 returned error can't find the container with id 79c8b2c2aa889a93498110b90fb71c8863ae1defe89791b08610e2528d1cc6d6
Jul 30 11:47:59.542237 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.542221    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 11:47:59.953554 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:47:59.953517    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ms7b/must-gather-958pc" event={"ID":"1ee3895e-e740-40d8-aa6e-770b66f759b4","Type":"ContainerStarted","Data":"79c8b2c2aa889a93498110b90fb71c8863ae1defe89791b08610e2528d1cc6d6"}
Jul 30 11:48:00.964404 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:00.963458    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ms7b/must-gather-958pc" event={"ID":"1ee3895e-e740-40d8-aa6e-770b66f759b4","Type":"ContainerStarted","Data":"475ee44ad4541c925191f7857868c3e3177f1eb1567597b89fe583ff2b9d0e24"}
Jul 30 11:48:01.980303 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:01.980224    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ms7b/must-gather-958pc" event={"ID":"1ee3895e-e740-40d8-aa6e-770b66f759b4","Type":"ContainerStarted","Data":"e7c00414869b0c7a11b6c9933c02ec5e53aaa6d2a07844ff05ccdf8adc72b1c2"}
Jul 30 11:48:02.000367 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:02.000317    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6ms7b/must-gather-958pc" podStartSLOduration=1.692058625 podStartE2EDuration="3.000300853s" totalImagesPullingTime="1.308242228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:48:01.999003132 +0000 UTC m=+3587.719887396" watchObservedRunningTime="2026-07-30 11:48:02.000300853 +0000 UTC m=+3587.721185114"
Jul 30 11:48:02.448842 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:02.448806    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cbrck_20d1cebf-7571-4c3b-b4fa-c6f6014a7e2d/global-pull-secret-syncer/0.log"
Jul 30 11:48:02.549959 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:02.549925    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log"
Jul 30 11:48:02.641030 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:02.640979    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-243.ec2.internal_db8ff15800a49eb6cf75db11d8895b81/haproxy/0.log"
Jul 30 11:48:07.708740 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:07.708301    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f6748df59-rb6zl_fe25eefe-9d08-4719-a944-7859dcf49aff/authorino/0.log"
Jul 30 11:48:07.798942 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:07.798901    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-j6kf2_1f1ee5b9-b701-4c17-8eb0-6b1c55b22672/kuadrant-console-plugin/0.log"
Jul 30 11:48:07.924774 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:07.924741    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-sthk2_fc0d75c4-8753-48f0-8104-9bc0e2304a55/limitador/0.log"
Jul 30 11:48:09.981195 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:09.981166    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvmqd_77f44fe7-44bd-404c-9845-bb065387ce70/node-exporter/0.log"
Jul 30 11:48:10.003523 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:10.003494    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvmqd_77f44fe7-44bd-404c-9845-bb065387ce70/kube-rbac-proxy/0.log"
Jul 30 11:48:10.029440 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:10.029164    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvmqd_77f44fe7-44bd-404c-9845-bb065387ce70/init-textfile/0.log"
Jul 30 11:48:12.174450 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.174395    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"]
Jul 30 11:48:12.177721 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.177694    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.186957 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.186751    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"]
Jul 30 11:48:12.192854 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.192819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-lib-modules\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.193001 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.192887    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lpz6\" (UniqueName: \"kubernetes.io/projected/f9dd7488-15f7-4a67-a138-28cb3fd81b60-kube-api-access-7lpz6\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.193001 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.192930    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-proc\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.193001 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.192956    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-sys\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.193001 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.192987    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-podres\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.293352 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.293313    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-lib-modules\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.293536 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.293378    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpz6\" (UniqueName: \"kubernetes.io/projected/f9dd7488-15f7-4a67-a138-28cb3fd81b60-kube-api-access-7lpz6\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.293536 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.293411    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-proc\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.293536 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.293429    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-sys\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.293536 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.293458    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-podres\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.293536 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.293497    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-lib-modules\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.293812 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.293534    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-proc\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.293812 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.293548    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-sys\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.293812 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.293585    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9dd7488-15f7-4a67-a138-28cb3fd81b60-podres\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.310346 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.310311    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lpz6\" (UniqueName: \"kubernetes.io/projected/f9dd7488-15f7-4a67-a138-28cb3fd81b60-kube-api-access-7lpz6\") pod \"perf-node-gather-daemonset-2xtp7\" (UID: \"f9dd7488-15f7-4a67-a138-28cb3fd81b60\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.492182 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.491963    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:12.682566 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:12.682494    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"]
Jul 30 11:48:12.689594 ip-10-0-128-243 kubenswrapper[2571]: W0730 11:48:12.689492    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9dd7488_15f7_4a67_a138_28cb3fd81b60.slice/crio-6a152fb9c96278c5802aaf0cae2d15c6e583c14d5d80aa8984e57ab959825032 WatchSource:0}: Error finding container 6a152fb9c96278c5802aaf0cae2d15c6e583c14d5d80aa8984e57ab959825032: Status 404 returned error can't find the container with id 6a152fb9c96278c5802aaf0cae2d15c6e583c14d5d80aa8984e57ab959825032
Jul 30 11:48:13.046077 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:13.045982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7" event={"ID":"f9dd7488-15f7-4a67-a138-28cb3fd81b60","Type":"ContainerStarted","Data":"6f314ec8f7ddc2da8e454696cdaef6a5b96ce78e85bcff1408a892a3b6cc9466"}
Jul 30 11:48:13.046077 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:13.046056    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7" event={"ID":"f9dd7488-15f7-4a67-a138-28cb3fd81b60","Type":"ContainerStarted","Data":"6a152fb9c96278c5802aaf0cae2d15c6e583c14d5d80aa8984e57ab959825032"}
Jul 30 11:48:13.046327 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:13.046132    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:14.756735 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:14.756701    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4swtg_8aeeb651-2172-4d17-bcb5-4e78a6b84278/dns/0.log"
Jul 30 11:48:14.775032 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:14.775003    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4swtg_8aeeb651-2172-4d17-bcb5-4e78a6b84278/kube-rbac-proxy/0.log"
Jul 30 11:48:14.903008 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:14.902961    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-grqcq_90bb7436-3324-44f3-8e91-c4a972c6c38f/dns-node-resolver/0.log"
Jul 30 11:48:15.551397 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:15.551363    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wvnjw_5df252dc-1dc8-40e4-a5f9-465fc2ee5819/node-ca/0.log"
Jul 30 11:48:16.661424 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:16.661396    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8674569cb6-mc94c_de38d89b-a26e-4d4d-999f-00a1585e9484/kube-auth-proxy/0.log"
Jul 30 11:48:17.677768 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:17.677736    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mzf92_1bdbeb13-8dfd-42e8-97bc-f692b86db785/serve-healthcheck-canary/0.log"
Jul 30 11:48:18.278976 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:18.278944    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-spcnq_5e588b42-d61a-4087-8a87-8b941a382ad2/kube-rbac-proxy/0.log"
Jul 30 11:48:18.296208 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:18.296178    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-spcnq_5e588b42-d61a-4087-8a87-8b941a382ad2/exporter/0.log"
Jul 30 11:48:18.314810 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:18.314781    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-spcnq_5e588b42-d61a-4087-8a87-8b941a382ad2/extractor/0.log"
Jul 30 11:48:19.063621 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:19.063587    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7"
Jul 30 11:48:19.084301 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:19.084234    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-2xtp7" podStartSLOduration=7.084213383 podStartE2EDuration="7.084213383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 11:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:48:13.06460522 +0000 UTC m=+3598.785489482" watchObservedRunningTime="2026-07-30 11:48:19.084213383 +0000 UTC m=+3604.805097647"
Jul 30 11:48:20.318090 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:20.318049    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4zj_d2a27b06-fcf7-4cbf-a646-ee312bada871/konnectivity-agent/0.log" limit=1024
Jul 30 11:48:21.235418 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:21.235382    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d97895fdd-4dqwm_cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba/manager/0.log"
Jul 30 11:48:21.254521 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:21.254492    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d97895fdd-4dqwm_cfb1a3d1-f6b0-4bbb-af9b-8d9b14e574ba/copy-manifests/0.log"
Jul 30 11:48:22.729996 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:22.729963    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-866774c595-tv5rb_32a98e99-6467-42f3-9c62-e73f43e48a4b/manager/0.log"
Jul 30 11:48:22.857003 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:22.856971    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8674569cb6-mc94c_de38d89b-a26e-4d4d-999f-00a1585e9484/kube-auth-proxy/0.log"
Jul 30 11:48:24.412454 ip-10-0-128-243 kubenswrapper[2571]: I0730 11:48:24.412420    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8674569cb6-mc94c_de38d89b-a26e-4d4d-999f-00a1585e9484/kube-auth-proxy/0.log"