Sep 16 04:46:55.741313 ip-10-0-133-117 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 04:46:55.741470 ip-10-0-133-117 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 04:46:55.741557 ip-10-0-133-117 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 04:46:55.741840 ip-10-0-133-117 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 04:47:06.855518 ip-10-0-133-117 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 04:47:06.855533 ip-10-0-133-117 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6fc79b96ec0d418b98a7f011563dc198 --
Sep 16 04:48:56.233880 ip-10-0-133-117 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 04:48:56.600361 ip-10-0-133-117 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 04:48:56.600361 ip-10-0-133-117 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 04:48:56.600361 ip-10-0-133-117 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 04:48:56.600361 ip-10-0-133-117 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 04:48:56.600361 ip-10-0-133-117 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 04:48:56.602559 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.602491    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 04:48:56.606075 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606060    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 04:48:56.606075 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606074    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606078    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606081    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606084    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606087    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606090    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606093    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606096    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606098    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606102    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606105    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606107    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606111    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606113    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606116    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606119    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606121    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606124    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606126    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606129    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 04:48:56.606133 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606131    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606134    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606136    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606139    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606142    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606145    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606148    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606150    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606153    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606155    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606158    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606161    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606163    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606167    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606170    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606173    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606176    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606178    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606181    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606184    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 04:48:56.606613 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606186    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606192    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606195    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606197    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606200    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606202    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606205    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606207    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606210    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606212    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606215    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606217    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606219    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606222    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606226    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606228    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606231    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606234    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606237    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 04:48:56.607076 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606240    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606243    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606245    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606248    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606250    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606253    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606255    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606258    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606260    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606262    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606265    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606270    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606273    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606276    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606279    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606284    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606287    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606291    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606294    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 04:48:56.607517 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606297    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606299    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606302    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606305    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606307    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606310    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606312    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606314    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606317    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606319    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606322    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606324    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606327    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606329    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606332    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606335    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606337    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606340    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606342    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.606345    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 04:48:56.607986 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607030    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607037    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607039    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607042    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607045    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607048    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607051    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607054    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607057    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607060    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607063    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607066    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607068    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607071    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607073    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607076    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607079    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607082    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607085    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607088    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 04:48:56.608463 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607090    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607093    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607096    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607098    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607101    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607103    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607106    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607108    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607111    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607114    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607116    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607119    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607122    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607124    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607127    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607129    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607132    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607135    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607138    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607140    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 04:48:56.608972 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607143    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607145    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607149    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607154    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607158    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607160    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607163    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607166    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607169    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607171    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607174    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607176    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607179    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607182    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607185    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607187    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607190    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607193    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607195    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 04:48:56.609448 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607198    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607200    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607203    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607205    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607208    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607211    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607214    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607216    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607219    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607223    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607225    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607228    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607230    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607233    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607235    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607238    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607241    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607244    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607246    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607249    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 04:48:56.609930 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607251    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607255    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607257    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607260    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607262    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607266    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607270    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607273    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607277    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607280    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607282    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607285    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607288    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607292    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607295    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607298    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607301    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607304    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607307    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 04:48:56.610398 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607309    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607687    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607693    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607697    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607701    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607705    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607708    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607711    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607714    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607718    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607721    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607724    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607727    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607729    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607732    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607735    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607738    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607741    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607743    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 04:48:56.610864 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607746    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607748    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607751    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607753    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607756    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607758    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607761    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607763    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607766    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607769    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607771    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607774    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607776    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607779    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607781    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607785    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607788    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607790    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607793    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607796    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 04:48:56.611330 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607798    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607801    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607804    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607807    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607810    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607812    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607814    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607817    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607819    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607822    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607824    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607827    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607829    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607831    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607834    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607837    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607839    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607842    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607844    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607846    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 04:48:56.611810 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607849    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607851    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607854    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607856    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607859    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607861    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607864    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607867    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607869    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607872    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607874    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607877    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607879    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607883    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607886    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607889    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607892    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607894    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607897    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 04:48:56.612290 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607899    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607902    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607904    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607907    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607909    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607912    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607914    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607917    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607919    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607922    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607924    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607927    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607931    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607935    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607941    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607944    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607947    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607949    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607952    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 04:48:56.612785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607955    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607958    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.607961    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608028    2573 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608071    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608078    2573 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608082    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608089    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608092    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608096    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608100    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608103    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608106    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608110    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608113    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608116    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608119    2573 flags.go:64] FLAG: --cgroup-root=""
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608121    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608124    2573 flags.go:64] FLAG: --client-ca-file=""
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608127    2573 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608130    2573 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608134    2573 flags.go:64] FLAG: --cluster-domain=""
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608137    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608140    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 04:48:56.613218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608143    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608148    2573 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608152    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608155    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608158    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608162    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608165    2573 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608167    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608170    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608174    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608177    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608181    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608184    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608187    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608190    2573 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608192    2573 flags.go:64] FLAG: --enable-server="true"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608196    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608200    2573 flags.go:64] FLAG: --event-burst="100"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608203    2573 flags.go:64] FLAG: --event-qps="50"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608206    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608209    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608212    2573 flags.go:64] FLAG: --eviction-hard=""
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608216    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608219    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608221    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 04:48:56.613801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608224    2573 flags.go:64] FLAG: --eviction-soft=""
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608228    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608231    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608233    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608236    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608239    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608242    2573 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608245    2573 flags.go:64] FLAG: --feature-gates=""
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608250    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608253    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608256    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608259    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608262    2573 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608266    2573 flags.go:64] FLAG: --help="false"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608268    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-133-117.ec2.internal"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608274    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608277    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608280    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608284    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608287    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608290    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608293    2573 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608295    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 04:48:56.614454 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608298    2573 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608301    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608304    2573 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608307    2573 flags.go:64] FLAG: --kube-reserved=""
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608310    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608313    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608316    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608319    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608322    2573 flags.go:64] FLAG: --lock-file=""
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608324    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608327    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608330    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608335    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608338    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608341    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608344    2573 flags.go:64] FLAG: --logging-format="text"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608346    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608354    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608357    2573 flags.go:64] FLAG: --manifest-url=""
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608359    2573 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608363    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608366    2573 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608370    2573 flags.go:64] FLAG: --max-pods="110"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608373    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608378    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 04:48:56.615063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608381    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608384    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608387    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608390    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608393    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608400    2573 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608404    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608407    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608410    2573 flags.go:64] FLAG: --pod-cidr=""
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608413    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608417    2573 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608420    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608423    2573 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608427    2573 flags.go:64] FLAG: --port="10250"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608430    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608433    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c76ba72e4e706175"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608436    2573 flags.go:64] FLAG: --qos-reserved=""
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608439    2573 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608442    2573 flags.go:64] FLAG: --register-node="true"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608445    2573 flags.go:64] FLAG: --register-with-taints=""
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608448    2573 flags.go:64] FLAG: --registry-burst="10"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608451    2573 flags.go:64] FLAG: --registry-qps="5"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608454    2573 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608457    2573 flags.go:64] FLAG: --reserved-memory=""
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608461    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 04:48:56.615704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608464    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608467    2573 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608470    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608473    2573 flags.go:64] FLAG: --runonce="false"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608476    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608479    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608483    2573 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608487    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608490    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608493    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608496    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608499    2573 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608502    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608505    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608508    2573 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608510    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608513    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608516    2573 flags.go:64] FLAG: --system-cgroups=""
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608519    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608524    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608527    2573 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608530    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608534    2573 flags.go:64] FLAG: --tls-min-version=""
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608537    2573 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608540    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 04:48:56.616384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608542    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608545    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608548    2573 flags.go:64] FLAG: --v="2"
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608552    2573 flags.go:64] FLAG: --version="false"
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608556    2573 flags.go:64] FLAG: --vmodule=""
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608560    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.608564    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608684    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608688    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608691    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608694    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608697    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608699    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608702    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608707    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608710    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608712    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608715    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608717    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608720    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608723    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 04:48:56.617345 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608725    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608728    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608731    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608733    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608736    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608739    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608743    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608746    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608750    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608753    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608756    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608758    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608761    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608764    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608767    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608770    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608772    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608777    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608779    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 04:48:56.617933 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608782    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608785    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608788    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608791    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608793    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608796    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608800    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608803    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608805    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608808    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608811    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608813    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608816    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608819    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608822    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608824    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608827    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608829    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608832    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608834    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608837    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 04:48:56.618391 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608840    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608843    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608845    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608848    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608850    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608853    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608855    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608858    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608860    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608864    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608867    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608869    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608872    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608874    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608877    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608880    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608882    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608886    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608889    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608892    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 04:48:56.618918 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608894    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608897    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608900    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608902    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608905    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608908    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608910    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608913    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608915    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608918    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608921    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608924    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608926    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608930    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608934    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608937    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608940    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608944    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608946    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 04:48:56.619401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608949    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608952    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608955    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608958    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608960    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.608964    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.609476    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.618168    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.618277    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.618297    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.618303    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618370    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618375    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618378    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 04:48:56.619973 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618382    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618385    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618388    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618391    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618394    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618397    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618400    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618403    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618406    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618409    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618411    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618415    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618419    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618422    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618425    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618428    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618431    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618434    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618436    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 04:48:56.620312 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618439    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618442    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618445    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618448    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618451    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618454    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618457    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618462    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618466    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618469    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618472    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618475    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618478    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618481    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618484    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618486    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618489    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618492    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618495    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618497    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 04:48:56.620785 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618500    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618502    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618505    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618508    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618511    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618514    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618517    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618520    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618522    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618525    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618527    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618530    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618532    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618535    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618537    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618540    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618543    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618545    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618548    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618552    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 04:48:56.621291 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618555    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618559    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618562    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618565    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618568    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618571    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618574    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618576    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618579    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618581    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618584    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618587    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618590    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618611    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618615    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618618    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618621    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618624    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618627    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 04:48:56.621793 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618630    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618632    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618635    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618637    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618640    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618642    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618645    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618647    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618650    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618652    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618655    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618657    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618660    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618663    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618667    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618669    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618672    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 04:48:56.622241 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618675    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.618680    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618792    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618797    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618800    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618804    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618806    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618810    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618814    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618817    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618820    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618822    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618825    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618828    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618831    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 04:48:56.622653 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618834    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618837    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618839    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618842    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618844    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618847    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618850    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618852    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618855    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618857    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618860    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618862    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618865    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618868    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618870    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618873    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618876    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618878    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618881    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618883    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 04:48:56.623003 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618886    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618888    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618891    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618893    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618895    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618898    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618901    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618903    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618906    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618908    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618910    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618913    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618916    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618918    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618921    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618924    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618927    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618929    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618932    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618935    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 04:48:56.623471 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618937    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618940    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618942    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618946    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618949    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618952    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618955    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618958    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618961    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618964    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618966    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618969    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618971    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618974    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618977    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618979    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618981    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618984    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618986    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 04:48:56.624021 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618989    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618992    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618995    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.618997    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619000    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619003    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619006    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619009    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619012    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619014    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619017    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619020    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619022    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619025    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619027    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619031    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619035    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619038    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619041    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 04:48:56.624491 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619044    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619046    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619049    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619052    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619054    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619057    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619059    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:56.619062    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.619066    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.619164    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.621326    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.622253    2573 server.go:1019] "Starting client certificate rotation"
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.622349    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 04:48:56.624951 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.623074    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 04:48:56.625265 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.624304    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 04:48:56.627553 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.627535    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 04:48:56.639686 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.639670    2573 log.go:25] "Validated CRI v1 runtime API"
Sep 16 04:48:56.643698 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.643685    2573 log.go:25] "Validated CRI v1 image API"
Sep 16 04:48:56.644528 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.644513    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 04:48:56.647826 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.647805    2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9b50a708-7716-4d11-9b02-4018c23de1d0:/dev/nvme0n1p4 c5e30f86-3fcd-4729-9af6-49f5913e7660:/dev/nvme0n1p3]
Sep 16 04:48:56.647898 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.647824    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 04:48:56.648394 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.648375    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 04:48:56.652668 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.652542    2573 manager.go:216] Machine: {Timestamp:2026-09-16 04:48:56.652083186 +0000 UTC m=+0.318012768 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3190947 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2007e59f1dc1aec44c65e5c9865078 SystemUUID:ec2007e5-9f1d-c1ae-c44c-65e5c9865078 BootID:6fc79b96-ec0d-418b-98a7-f011563dc198 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e7:ad:c2:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e7:ad:c2:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:0b:fd:3b:22:0f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 04:48:56.652668 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.652653    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 04:48:56.652779 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.652747    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 04:48:56.654232 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.654208    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 04:48:56.654351 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.654234    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 04:48:56.654858 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.654848    2573 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 04:48:56.654894 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.654861    2573 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 04:48:56.654894 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.654874    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 04:48:56.655458 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.655448    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 04:48:56.656159 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.656148    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 04:48:56.659826 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.659815    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 04:48:56.659935 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.659926    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 04:48:56.659969 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.659955    2573 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 04:48:56.659969 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.659963    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 04:48:56.660018 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.659977    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 04:48:56.660018 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.659985    2573 kubelet.go:398] "Adding apiserver pod source"
Sep 16 04:48:56.660018 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.659993    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 04:48:56.660295 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.660283    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 04:48:56.660359 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.660301    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 04:48:56.662625 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.662606    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 04:48:56.664228 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.664215    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 04:48:56.664275 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.664244    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 04:48:56.666210 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666196    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 04:48:56.666278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666214    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 04:48:56.666278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666221    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 04:48:56.666278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666226    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 04:48:56.666278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666235    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 04:48:56.666278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666241    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 04:48:56.666278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666253    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 04:48:56.666278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666258    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 04:48:56.666278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666265    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 04:48:56.666278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666270    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 04:48:56.666278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666279    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 04:48:56.666524 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.666284    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 04:48:56.667035 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.667025    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 04:48:56.667035 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.667035    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 04:48:56.667986 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.667957    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 04:48:56.667986 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.667969    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 04:48:56.670356 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.670340    2573 server.go:1268] "Started kubelet"
Sep 16 04:48:56.670467 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.670421    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 04:48:56.670515 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.670492    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 04:48:56.671018 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.670985    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 04:48:56.671119 ip-10-0-133-117 systemd[1]: Started Kubernetes Kubelet.
Sep 16 04:48:56.671392 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.671378    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 04:48:56.672934 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.672920    2573 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 04:48:56.675071 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.674560    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 04:48:56.678387 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.678368    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 04:48:56.678806 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.678789    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 04:48:56.679058 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679010    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 04:48:56.679114 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.679039    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:56.679461 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.678554    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-117.ec2.internal.18d5b3d0e1ab5ec9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-117.ec2.internal,UID:ip-10-0-133-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-117.ec2.internal,},FirstTimestamp:2026-09-16 04:48:56.670314185 +0000 UTC m=+0.336243768,LastTimestamp:2026-09-16 04:48:56.670314185 +0000 UTC m=+0.336243768,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-117.ec2.internal,}"
Sep 16 04:48:56.679700 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679649    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 04:48:56.679700 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679674    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 04:48:56.679700 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679691    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 04:48:56.679871 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679768    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 04:48:56.679871 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679777    2573 factory.go:55] Registering systemd factory
Sep 16 04:48:56.679871 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679783    2573 factory.go:223] Registration of the systemd container factory successfully
Sep 16 04:48:56.679871 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679808    2573 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 04:48:56.679871 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679818    2573 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 04:48:56.680003 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679954    2573 factory.go:153] Registering CRI-O factory
Sep 16 04:48:56.680003 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679965    2573 factory.go:223] Registration of the crio container factory successfully
Sep 16 04:48:56.680003 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679978    2573 factory.go:103] Registering Raw factory
Sep 16 04:48:56.680003 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.679987    2573 manager.go:1196] Started watching for new ooms in manager
Sep 16 04:48:56.680344 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.680333    2573 manager.go:349] Starting recovery of all containers
Sep 16 04:48:56.681066 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.681024    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 04:48:56.681150 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.681134    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 04:48:56.686031 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.685871    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pfbpp"
Sep 16 04:48:56.689071 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.689029    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 04:48:56.691092 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.691074    2573 manager.go:354] Recovery completed
Sep 16 04:48:56.691856 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.691834    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pfbpp"
Sep 16 04:48:56.695283 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.695272    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 04:48:56.697333 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.697318    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 04:48:56.697402 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.697348    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 04:48:56.697402 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.697360    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientPID"
Sep 16 04:48:56.697863 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.697849    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 04:48:56.697863 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.697862    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 04:48:56.697946 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.697875    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 04:48:56.697946 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.697907    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 04:48:56.698812 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.698798    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 04:48:56.698864 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.698817    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 04:48:56.698864 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.698830    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientPID"
Sep 16 04:48:56.699377 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.699318    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-117.ec2.internal.18d5b3d0e3479f01  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-117.ec2.internal,UID:ip-10-0-133-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-117.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-117.ec2.internal,},FirstTimestamp:2026-09-16 04:48:56.697331457 +0000 UTC m=+0.363261043,LastTimestamp:2026-09-16 04:48:56.697331457 +0000 UTC m=+0.363261043,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-117.ec2.internal,}"
Sep 16 04:48:56.699934 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.699921    2573 policy_none.go:49] "None policy: Start"
Sep 16 04:48:56.700005 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.699938    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 04:48:56.700005 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.699951    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 04:48:56.702000 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.701988    2573 policy_none.go:47] "Start"
Sep 16 04:48:56.727269 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.727257    2573 manager.go:337] "Starting Device Plugin manager"
Sep 16 04:48:56.727342 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.727294    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 04:48:56.727342 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.727306    2573 server.go:87] "Starting device plugin registration server"
Sep 16 04:48:56.727529 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.727516    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 04:48:56.727652 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.727530    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 04:48:56.727652 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.727634    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 04:48:56.727760 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.727703    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 04:48:56.727760 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.727711    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 04:48:56.728190 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.728131    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 04:48:56.728265 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.728215    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.779507    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.779550    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.779566    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.779615    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.781871    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.828055    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.828779    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.828804    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.828814    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientPID"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.828832    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-117.ec2.internal"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.837501    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-117.ec2.internal"
Sep 16 04:48:56.846219 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.837517    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-117.ec2.internal\": node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:56.850482 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.850428    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:56.880539 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.880520    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal"]
Sep 16 04:48:56.880623 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.880572    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 04:48:56.881802 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.881787    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 04:48:56.881854 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.881814    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 04:48:56.881854 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.881824    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientPID"
Sep 16 04:48:56.884018 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.884006    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 04:48:56.884128 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.884114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:56.884166 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.884146    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 04:48:56.884656 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.884642    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 04:48:56.884710 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.884662    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 04:48:56.884710 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.884670    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientPID"
Sep 16 04:48:56.884710 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.884703    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 04:48:56.884801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.884721    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 04:48:56.884801 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.884731    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientPID"
Sep 16 04:48:56.886756 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.886743    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:56.886803 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.886765    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 04:48:56.887344 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.887322    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 04:48:56.887384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.887348    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 04:48:56.887384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.887359    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeHasSufficientPID"
Sep 16 04:48:56.905170 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.905155    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-117.ec2.internal\" not found" node="ip-10-0-133-117.ec2.internal"
Sep 16 04:48:56.909303 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.909288    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-117.ec2.internal\" not found" node="ip-10-0-133-117.ec2.internal"
Sep 16 04:48:56.951137 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:56.951123    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:56.980919 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.980902    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d4e64cb672ced1a149e7f33c599dbd9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal\" (UID: \"4d4e64cb672ced1a149e7f33c599dbd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:56.980994 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.980923    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d4e64cb672ced1a149e7f33c599dbd9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal\" (UID: \"4d4e64cb672ced1a149e7f33c599dbd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:56.980994 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:56.980938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/461b5598d7024278d7832d79de8a2c73-config\") pod \"kube-apiserver-proxy-ip-10-0-133-117.ec2.internal\" (UID: \"461b5598d7024278d7832d79de8a2c73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:57.051738 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:57.051711    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:57.082056 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.082042    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d4e64cb672ced1a149e7f33c599dbd9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal\" (UID: \"4d4e64cb672ced1a149e7f33c599dbd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:57.082116 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.082063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d4e64cb672ced1a149e7f33c599dbd9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal\" (UID: \"4d4e64cb672ced1a149e7f33c599dbd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:57.082116 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.082080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/461b5598d7024278d7832d79de8a2c73-config\") pod \"kube-apiserver-proxy-ip-10-0-133-117.ec2.internal\" (UID: \"461b5598d7024278d7832d79de8a2c73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:57.082184 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.082133    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/461b5598d7024278d7832d79de8a2c73-config\") pod \"kube-apiserver-proxy-ip-10-0-133-117.ec2.internal\" (UID: \"461b5598d7024278d7832d79de8a2c73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:57.082184 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.082152    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d4e64cb672ced1a149e7f33c599dbd9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal\" (UID: \"4d4e64cb672ced1a149e7f33c599dbd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:57.082184 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.082152    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d4e64cb672ced1a149e7f33c599dbd9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal\" (UID: \"4d4e64cb672ced1a149e7f33c599dbd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:57.152309 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:57.152272    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:57.209912 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.209895    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:57.212047 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.212035    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:57.253029 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:57.253012    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:57.353486 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:57.353457    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:57.454011 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:57.453964    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:57.554416 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:57.554398    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:57.621927 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.621902    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 04:48:57.622349 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.622005    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 04:48:57.655180 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:57.655165    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:57.696655 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.696506    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 04:43:57 +0000 UTC" deadline="2028-02-24 09:48:06.171480069 +0000 UTC"
Sep 16 04:48:57.696655 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.696655    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12628h59m8.474829194s"
Sep 16 04:48:57.755768 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:57.755749    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:57.775209 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:57.775171    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4e64cb672ced1a149e7f33c599dbd9.slice/crio-bb458cee24a91079cbea7c1c31a760f0442a7e112359becad852eef3c4297cfc WatchSource:0}: Error finding container bb458cee24a91079cbea7c1c31a760f0442a7e112359becad852eef3c4297cfc: Status 404 returned error can't find the container with id bb458cee24a91079cbea7c1c31a760f0442a7e112359becad852eef3c4297cfc
Sep 16 04:48:57.775434 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:57.775412    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461b5598d7024278d7832d79de8a2c73.slice/crio-7f615bcdea5e5c41c778fa2b376b79032e0cf9d76f82056fd5d0f1cbf980f049 WatchSource:0}: Error finding container 7f615bcdea5e5c41c778fa2b376b79032e0cf9d76f82056fd5d0f1cbf980f049: Status 404 returned error can't find the container with id 7f615bcdea5e5c41c778fa2b376b79032e0cf9d76f82056fd5d0f1cbf980f049
Sep 16 04:48:57.779461 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.779441    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 04:48:57.783675 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.783628    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal" event={"ID":"4d4e64cb672ced1a149e7f33c599dbd9","Type":"ContainerStarted","Data":"bb458cee24a91079cbea7c1c31a760f0442a7e112359becad852eef3c4297cfc"}
Sep 16 04:48:57.784922 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.784898    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal" event={"ID":"461b5598d7024278d7832d79de8a2c73","Type":"ContainerStarted","Data":"7f615bcdea5e5c41c778fa2b376b79032e0cf9d76f82056fd5d0f1cbf980f049"}
Sep 16 04:48:57.790453 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.790436    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 04:48:57.858949 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:57.858918    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:57.898674 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:57.898651    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 04:48:57.962315 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:57.962292    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:58.065941 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.065894    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:58.166109 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.166083    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-117.ec2.internal\" not found"
Sep 16 04:48:58.250342 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.250322    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 04:48:58.279579 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.279556    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:58.291166 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.291149    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 04:48:58.292120 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.292103    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal"
Sep 16 04:48:58.300504 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.300481    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 04:48:58.661314 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.661280    2573 apiserver.go:52] "Watching apiserver"
Sep 16 04:48:58.668003 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.667976    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 04:48:58.668449 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.668422    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-24mv8","kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw","openshift-image-registry/node-ca-rw649","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal","openshift-multus/multus-78g6m","openshift-multus/multus-additional-cni-plugins-2tvkn","openshift-network-operator/iptables-alerter-zrwk4","kube-system/global-pull-secret-syncer-vqmw5","openshift-cluster-node-tuning-operator/tuned-5cpkt","openshift-dns/node-resolver-69szj","openshift-multus/network-metrics-daemon-k449w","openshift-network-diagnostics/network-check-target-64c4f","openshift-ovn-kubernetes/ovnkube-node-w4q88"]
Sep 16 04:48:58.671916 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.671190    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.673739 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.673713    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:58.674346 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.674124    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 04:48:58.674441 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.674404    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hg57j\""
Sep 16 04:48:58.674510 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.674471    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 04:48:58.674623 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.674602    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 04:48:58.674728 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.674710    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 04:48:58.674821 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.674805    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 04:48:58.675907 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.675885    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 04:48:58.676019 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.675965    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.677451 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.677400    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 04:48:58.678398 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.678376    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 04:48:58.679537 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.678621    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rwdhj\""
Sep 16 04:48:58.679537 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.678816    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-txr2g\""
Sep 16 04:48:58.679537 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.678625    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 04:48:58.679537 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.679389    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 04:48:58.680108 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.680088    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:58.680931 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.680908    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 04:48:58.681986 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.681967    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 04:48:58.682469 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.682442    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6w2sw\""
Sep 16 04:48:58.682629 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.682600    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 04:48:58.682723 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.682603    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 04:48:58.684068 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.683463    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.685588 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.685566    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 04:48:58.685811 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.685792    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 04:48:58.685962 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.685794    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-46qtf\""
Sep 16 04:48:58.686084 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.686067    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 04:48:58.686169 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.686153    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 04:48:58.686282 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.686269    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 04:48:58.686866 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.686850    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 04:48:58.688299 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.688280    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.688881 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.688859    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 04:48:58.689039 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6985599b-f750-48f3-9e34-55dd74ed94bc-ovnkube-script-lib\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acab9663-a074-4fbb-8596-a50e5e39b878-system-cni-dir\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.689118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689065    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acab9663-a074-4fbb-8596-a50e5e39b878-cnibin\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.689118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acab9663-a074-4fbb-8596-a50e5e39b878-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.689249 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689150    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acab9663-a074-4fbb-8596-a50e5e39b878-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.689249 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689191    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acab9663-a074-4fbb-8596-a50e5e39b878-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.689249 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.689360 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689248    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-run-systemd\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689360 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-run-openvswitch\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689360 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689302    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-cni-netd\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689504 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689361    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689504 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689388    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689504 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689413    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64223cf5-19e6-4f4a-aa87-67766e0373d9-tmp-dir\") pod \"node-resolver-69szj\" (UID: \"64223cf5-19e6-4f4a-aa87-67766e0373d9\") " pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:58.689504 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689445    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qbl\" (UniqueName: \"kubernetes.io/projected/64223cf5-19e6-4f4a-aa87-67766e0373d9-kube-api-access-t9qbl\") pod \"node-resolver-69szj\" (UID: \"64223cf5-19e6-4f4a-aa87-67766e0373d9\") " pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:58.689504 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-slash\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689504 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689496    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6985599b-f750-48f3-9e34-55dd74ed94bc-env-overrides\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689804 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689521    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5fv\" (UniqueName: \"kubernetes.io/projected/6985599b-f750-48f3-9e34-55dd74ed94bc-kube-api-access-ms5fv\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689804 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689547    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acab9663-a074-4fbb-8596-a50e5e39b878-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.689804 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689572    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzm6t\" (UniqueName: \"kubernetes.io/projected/9d65b077-2a03-48e7-ba99-7bd68b698d75-kube-api-access-fzm6t\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.689804 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689595    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01b6aa0b-369b-4585-bcbc-890512014439-serviceca\") pod \"node-ca-rw649\" (UID: \"01b6aa0b-369b-4585-bcbc-890512014439\") " pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:58.689804 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689634    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-etc-openvswitch\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689804 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689658    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-log-socket\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.689804 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689693    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64223cf5-19e6-4f4a-aa87-67766e0373d9-hosts-file\") pod \"node-resolver-69szj\" (UID: \"64223cf5-19e6-4f4a-aa87-67766e0373d9\") " pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:58.689804 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.689804 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689752    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-socket-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.689804 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689776    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-node-log\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689798    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9pjf\" (UniqueName: \"kubernetes.io/projected/acab9663-a074-4fbb-8596-a50e5e39b878-kube-api-access-b9pjf\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689829    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-systemd-units\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-run-ovn\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689929    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-cni-bin\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6985599b-f750-48f3-9e34-55dd74ed94bc-ovnkube-config\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.689988    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-registration-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690106    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4gxvp\""
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-device-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690161    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-sys-fs\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhthg\" (UniqueName: \"kubernetes.io/projected/01b6aa0b-369b-4585-bcbc-890512014439-kube-api-access-bhthg\") pod \"node-ca-rw649\" (UID: \"01b6aa0b-369b-4585-bcbc-890512014439\") " pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-kubelet\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-run-netns\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.690245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690246    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 04:48:58.690877 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-var-lib-openvswitch\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.690877 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690281    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01b6aa0b-369b-4585-bcbc-890512014439-host\") pod \"node-ca-rw649\" (UID: \"01b6aa0b-369b-4585-bcbc-890512014439\") " pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:58.690877 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690306    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6985599b-f750-48f3-9e34-55dd74ed94bc-ovn-node-metrics-cert\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.690877 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.690581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:48:58.690877 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.690679    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:48:58.693898 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.693876    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:48:58.693998 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.693940    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:48:58.696784 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.696763    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:58.699733 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.699410    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.700115 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.700094    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 04:48:58.700221 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.700190    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 04:48:58.700282 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.700096    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lxrh7\""
Sep 16 04:48:58.700333 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.700285    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 04:48:58.701213 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.701193    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 04:48:58.701369 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.701348    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 04:48:58.701445 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.701426    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lpz4r\""
Sep 16 04:48:58.701936 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.701919    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:48:58.703844 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.703809    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 04:48:58.704810 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.704751    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 04:48:58.704907 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.704833    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qprn5\""
Sep 16 04:48:58.705190 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.705170    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:48:58.705336 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.705235    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:48:58.714323 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.714300    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mrws2"
Sep 16 04:48:58.723169 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.723150    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mrws2"
Sep 16 04:48:58.780788 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.780755    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 04:48:58.790546 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.790681 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790560    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-run-openvswitch\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.790681 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790588    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-cni-netd\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.790681 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790634    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-sys\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.790681 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790662    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-var-lib-kubelet\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.790857 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790734    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-cni-netd\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.790857 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790812    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-run-openvswitch\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.790985 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790853    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.790985 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790856    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.790985 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790893    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.790985 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-kubernetes\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.790985 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790923    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b33becd7-76a1-4999-93f7-94222c51c7fd-tmp\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.790985 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5fv\" (UniqueName: \"kubernetes.io/projected/6985599b-f750-48f3-9e34-55dd74ed94bc-kube-api-access-ms5fv\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.790985 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.790976    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c097b7c-cb2b-4fec-844f-72d0f78a12fe-host-slash\") pod \"iptables-alerter-zrwk4\" (UID: \"9c097b7c-cb2b-4fec-844f-72d0f78a12fe\") " pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:58.791267 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791001    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-run-k8s-cni-cncf-io\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.791267 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791138    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-run-multus-certs\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.791267 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01b6aa0b-369b-4585-bcbc-890512014439-serviceca\") pod \"node-ca-rw649\" (UID: \"01b6aa0b-369b-4585-bcbc-890512014439\") " pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:58.791267 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-log-socket\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.791267 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791222    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-cni-dir\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.791267 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791247    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36a9d2ea-0102-4624-bfa6-b96860b1f483-agent-certs\") pod \"konnectivity-agent-24mv8\" (UID: \"36a9d2ea-0102-4624-bfa6-b96860b1f483\") " pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791297    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-node-log\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-systemd-units\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791372    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-run-ovn\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6985599b-f750-48f3-9e34-55dd74ed94bc-ovnkube-config\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791420    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-conf-dir\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791444    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-etc-kubernetes\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791469    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-sys-fs\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhthg\" (UniqueName: \"kubernetes.io/projected/01b6aa0b-369b-4585-bcbc-890512014439-kube-api-access-bhthg\") pod \"node-ca-rw649\" (UID: \"01b6aa0b-369b-4585-bcbc-890512014439\") " pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791553    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-kubelet\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.791594 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6985599b-f750-48f3-9e34-55dd74ed94bc-ovn-node-metrics-cert\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64223cf5-19e6-4f4a-aa87-67766e0373d9-tmp-dir\") pod \"node-resolver-69szj\" (UID: \"64223cf5-19e6-4f4a-aa87-67766e0373d9\") " pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qbl\" (UniqueName: \"kubernetes.io/projected/64223cf5-19e6-4f4a-aa87-67766e0373d9-kube-api-access-t9qbl\") pod \"node-resolver-69szj\" (UID: \"64223cf5-19e6-4f4a-aa87-67766e0373d9\") " pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791676    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c097b7c-cb2b-4fec-844f-72d0f78a12fe-iptables-alerter-script\") pod \"iptables-alerter-zrwk4\" (UID: \"9c097b7c-cb2b-4fec-844f-72d0f78a12fe\") " pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01b6aa0b-369b-4585-bcbc-890512014439-host\") pod \"node-ca-rw649\" (UID: \"01b6aa0b-369b-4585-bcbc-890512014439\") " pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791728    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6985599b-f750-48f3-9e34-55dd74ed94bc-ovnkube-script-lib\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791753    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acab9663-a074-4fbb-8596-a50e5e39b878-cnibin\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791778    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acab9663-a074-4fbb-8596-a50e5e39b878-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791804    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acab9663-a074-4fbb-8596-a50e5e39b878-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791838    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-system-cni-dir\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791867    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f784j\" (UniqueName: \"kubernetes.io/projected/b33becd7-76a1-4999-93f7-94222c51c7fd-kube-api-access-f784j\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791907    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-node-log\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791930    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791947    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-systemd-units\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.791968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-run-systemd\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792012    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-run-ovn\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792035    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-sys-fs\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.792118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792049    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01b6aa0b-369b-4585-bcbc-890512014439-host\") pod \"node-ca-rw649\" (UID: \"01b6aa0b-369b-4585-bcbc-890512014439\") " pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792071    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acab9663-a074-4fbb-8596-a50e5e39b878-cnibin\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792278    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-log-socket\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6985599b-f750-48f3-9e34-55dd74ed94bc-ovnkube-config\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792346    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64223cf5-19e6-4f4a-aa87-67766e0373d9-tmp-dir\") pod \"node-resolver-69szj\" (UID: \"64223cf5-19e6-4f4a-aa87-67766e0373d9\") " pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792442    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq2mc\" (UniqueName: \"kubernetes.io/projected/9c097b7c-cb2b-4fec-844f-72d0f78a12fe-kube-api-access-kq2mc\") pod \"iptables-alerter-zrwk4\" (UID: \"9c097b7c-cb2b-4fec-844f-72d0f78a12fe\") " pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792460    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-run-netns\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792483    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9705e01-c8d0-481f-8769-ffd767bc202c-dbus\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792562    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-kubelet\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792564    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6985599b-f750-48f3-9e34-55dd74ed94bc-ovnkube-script-lib\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792578    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsv87\" (UniqueName: \"kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87\") pod \"network-check-target-64c4f\" (UID: \"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5\") " pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792589    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acab9663-a074-4fbb-8596-a50e5e39b878-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792606    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b25e368-49cf-4c2e-840d-2347f6ceb400-cni-binary-copy\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792634    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mcl2\" (UniqueName: \"kubernetes.io/projected/7b25e368-49cf-4c2e-840d-2347f6ceb400-kube-api-access-7mcl2\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792686    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-sysconfig\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.792879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-systemd\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792752    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01b6aa0b-369b-4585-bcbc-890512014439-serviceca\") pod \"node-ca-rw649\" (UID: \"01b6aa0b-369b-4585-bcbc-890512014439\") " pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792754    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-slash\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792793    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-slash\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6985599b-f750-48f3-9e34-55dd74ed94bc-env-overrides\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792821    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acab9663-a074-4fbb-8596-a50e5e39b878-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792808    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-run-systemd\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acab9663-a074-4fbb-8596-a50e5e39b878-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-socket-dir-parent\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792906    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-var-lib-cni-bin\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792816    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792931    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlglz\" (UniqueName: \"kubernetes.io/projected/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-kube-api-access-jlglz\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792964    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzm6t\" (UniqueName: \"kubernetes.io/projected/9d65b077-2a03-48e7-ba99-7bd68b698d75-kube-api-access-fzm6t\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792981    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-etc-openvswitch\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.792984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acab9663-a074-4fbb-8596-a50e5e39b878-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793021    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64223cf5-19e6-4f4a-aa87-67766e0373d9-hosts-file\") pod \"node-resolver-69szj\" (UID: \"64223cf5-19e6-4f4a-aa87-67766e0373d9\") " pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793043    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-socket-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.793641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793048    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-etc-openvswitch\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793059    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9pjf\" (UniqueName: \"kubernetes.io/projected/acab9663-a074-4fbb-8596-a50e5e39b878-kube-api-access-b9pjf\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793087    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-hostroot\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793112    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-sysctl-conf\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793135    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-run\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793142    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64223cf5-19e6-4f4a-aa87-67766e0373d9-hosts-file\") pod \"node-resolver-69szj\" (UID: \"64223cf5-19e6-4f4a-aa87-67766e0373d9\") " pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36a9d2ea-0102-4624-bfa6-b96860b1f483-konnectivity-ca\") pod \"konnectivity-agent-24mv8\" (UID: \"36a9d2ea-0102-4624-bfa6-b96860b1f483\") " pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793188    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9705e01-c8d0-481f-8769-ffd767bc202c-kubelet-config\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-cni-bin\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793239    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-var-lib-kubelet\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793271    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-socket-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-modprobe-d\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793285    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6985599b-f750-48f3-9e34-55dd74ed94bc-env-overrides\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793313    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-cni-bin\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793366    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-host\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-tuned\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793431    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-registration-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.794327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793474    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-device-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793484    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-registration-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-run-netns\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793536    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d65b077-2a03-48e7-ba99-7bd68b698d75-device-dir\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793537    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-var-lib-openvswitch\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793558    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-host-run-netns\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793576    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6985599b-f750-48f3-9e34-55dd74ed94bc-var-lib-openvswitch\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acab9663-a074-4fbb-8596-a50e5e39b878-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-var-lib-cni-multus\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acab9663-a074-4fbb-8596-a50e5e39b878-system-cni-dir\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793689    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-cnibin\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793714    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-daemon-config\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793737    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-sysctl-d\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793743    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acab9663-a074-4fbb-8596-a50e5e39b878-system-cni-dir\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793762    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-lib-modules\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.793812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:48:58.795072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.794024    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acab9663-a074-4fbb-8596-a50e5e39b878-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.798053 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.798001    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6985599b-f750-48f3-9e34-55dd74ed94bc-ovn-node-metrics-cert\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.801406 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.801325    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9pjf\" (UniqueName: \"kubernetes.io/projected/acab9663-a074-4fbb-8596-a50e5e39b878-kube-api-access-b9pjf\") pod \"multus-additional-cni-plugins-2tvkn\" (UID: \"acab9663-a074-4fbb-8596-a50e5e39b878\") " pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.801498 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.801463    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qbl\" (UniqueName: \"kubernetes.io/projected/64223cf5-19e6-4f4a-aa87-67766e0373d9-kube-api-access-t9qbl\") pod \"node-resolver-69szj\" (UID: \"64223cf5-19e6-4f4a-aa87-67766e0373d9\") " pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:58.801498 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.801473    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5fv\" (UniqueName: \"kubernetes.io/projected/6985599b-f750-48f3-9e34-55dd74ed94bc-kube-api-access-ms5fv\") pod \"ovnkube-node-w4q88\" (UID: \"6985599b-f750-48f3-9e34-55dd74ed94bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:58.801997 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.801956    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzm6t\" (UniqueName: \"kubernetes.io/projected/9d65b077-2a03-48e7-ba99-7bd68b698d75-kube-api-access-fzm6t\") pod \"aws-ebs-csi-driver-node-b6lhw\" (UID: \"9d65b077-2a03-48e7-ba99-7bd68b698d75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:58.802261 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.802240    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhthg\" (UniqueName: \"kubernetes.io/projected/01b6aa0b-369b-4585-bcbc-890512014439-kube-api-access-bhthg\") pod \"node-ca-rw649\" (UID: \"01b6aa0b-369b-4585-bcbc-890512014439\") " pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:58.894178 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-conf-dir\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894289 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-etc-kubernetes\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894289 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894218    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c097b7c-cb2b-4fec-844f-72d0f78a12fe-iptables-alerter-script\") pod \"iptables-alerter-zrwk4\" (UID: \"9c097b7c-cb2b-4fec-844f-72d0f78a12fe\") " pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:58.894289 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894248    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-system-cni-dir\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894289 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894267    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-conf-dir\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894507 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894272    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f784j\" (UniqueName: \"kubernetes.io/projected/b33becd7-76a1-4999-93f7-94222c51c7fd-kube-api-access-f784j\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.894507 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894273    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-etc-kubernetes\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894507 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq2mc\" (UniqueName: \"kubernetes.io/projected/9c097b7c-cb2b-4fec-844f-72d0f78a12fe-kube-api-access-kq2mc\") pod \"iptables-alerter-zrwk4\" (UID: \"9c097b7c-cb2b-4fec-844f-72d0f78a12fe\") " pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:58.894507 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894407    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-run-netns\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894507 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894357    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-system-cni-dir\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894507 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9705e01-c8d0-481f-8769-ffd767bc202c-dbus\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:48:58.894507 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894467    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-run-netns\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894507 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894470    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv87\" (UniqueName: \"kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87\") pod \"network-check-target-64c4f\" (UID: \"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5\") " pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:48:58.894507 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b25e368-49cf-4c2e-840d-2347f6ceb400-cni-binary-copy\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894523    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mcl2\" (UniqueName: \"kubernetes.io/projected/7b25e368-49cf-4c2e-840d-2347f6ceb400-kube-api-access-7mcl2\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894550    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-sysconfig\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-systemd\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894604    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9705e01-c8d0-481f-8769-ffd767bc202c-dbus\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-socket-dir-parent\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-var-lib-cni-bin\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894640    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-sysconfig\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894677    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlglz\" (UniqueName: \"kubernetes.io/projected/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-kube-api-access-jlglz\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894708    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-socket-dir-parent\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-hostroot\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894753    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-systemd\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894755    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-hostroot\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894780    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-var-lib-cni-bin\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-sysctl-conf\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-run\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36a9d2ea-0102-4624-bfa6-b96860b1f483-konnectivity-ca\") pod \"konnectivity-agent-24mv8\" (UID: \"36a9d2ea-0102-4624-bfa6-b96860b1f483\") " pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894862    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9705e01-c8d0-481f-8769-ffd767bc202c-kubelet-config\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:48:58.894975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894880    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-run\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894861    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c097b7c-cb2b-4fec-844f-72d0f78a12fe-iptables-alerter-script\") pod \"iptables-alerter-zrwk4\" (UID: \"9c097b7c-cb2b-4fec-844f-72d0f78a12fe\") " pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894913    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9705e01-c8d0-481f-8769-ffd767bc202c-kubelet-config\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-var-lib-kubelet\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-modprobe-d\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-host\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894968    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-sysctl-conf\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.894989    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-tuned\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895026    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-var-lib-cni-multus\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895032    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-host\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895034    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-var-lib-kubelet\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895041    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b25e368-49cf-4c2e-840d-2347f6ceb400-cni-binary-copy\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895054    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-cnibin\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-daemon-config\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895098    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-var-lib-cni-multus\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-sysctl-d\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895120    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-modprobe-d\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895133    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-lib-modules\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.895842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895153    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-cnibin\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895158    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-sys\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-var-lib-kubelet\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.895245    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895285    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-kubernetes\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.895326    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret podName:b9705e01-c8d0-481f-8769-ffd767bc202c nodeName:}" failed. No retries permitted until 2026-09-16 04:48:59.395285469 +0000 UTC m=+3.061215055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret") pod "global-pull-secret-syncer-vqmw5" (UID: "b9705e01-c8d0-481f-8769-ffd767bc202c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-kubernetes\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895333    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-var-lib-kubelet\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b33becd7-76a1-4999-93f7-94222c51c7fd-tmp\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895322    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-sys\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895331    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36a9d2ea-0102-4624-bfa6-b96860b1f483-konnectivity-ca\") pod \"konnectivity-agent-24mv8\" (UID: \"36a9d2ea-0102-4624-bfa6-b96860b1f483\") " pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c097b7c-cb2b-4fec-844f-72d0f78a12fe-host-slash\") pod \"iptables-alerter-zrwk4\" (UID: \"9c097b7c-cb2b-4fec-844f-72d0f78a12fe\") " pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895428    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c097b7c-cb2b-4fec-844f-72d0f78a12fe-host-slash\") pod \"iptables-alerter-zrwk4\" (UID: \"9c097b7c-cb2b-4fec-844f-72d0f78a12fe\") " pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895439    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-sysctl-d\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-run-k8s-cni-cncf-io\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895469    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b33becd7-76a1-4999-93f7-94222c51c7fd-lib-modules\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.896654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895484    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-run-k8s-cni-cncf-io\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.897329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-run-multus-certs\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.897329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895520    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-host-run-multus-certs\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.897329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-cni-dir\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.897329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895548    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-daemon-config\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.897329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895553    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36a9d2ea-0102-4624-bfa6-b96860b1f483-agent-certs\") pod \"konnectivity-agent-24mv8\" (UID: \"36a9d2ea-0102-4624-bfa6-b96860b1f483\") " pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:48:58.897329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895608    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b25e368-49cf-4c2e-840d-2347f6ceb400-multus-cni-dir\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.897329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.895606    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:48:58.897329 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.895680    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:48:58.897329 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.895743    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs podName:835f4ee4-d6b0-4fb6-b6e6-86277d856c67 nodeName:}" failed. No retries permitted until 2026-09-16 04:48:59.39571661 +0000 UTC m=+3.061646197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs") pod "network-metrics-daemon-k449w" (UID: "835f4ee4-d6b0-4fb6-b6e6-86277d856c67") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:48:58.897725 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.897417    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b33becd7-76a1-4999-93f7-94222c51c7fd-etc-tuned\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.897986 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.897964    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36a9d2ea-0102-4624-bfa6-b96860b1f483-agent-certs\") pod \"konnectivity-agent-24mv8\" (UID: \"36a9d2ea-0102-4624-bfa6-b96860b1f483\") " pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:48:58.900352 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.900322    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b33becd7-76a1-4999-93f7-94222c51c7fd-tmp\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.902665 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.902645    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 04:48:58.902665 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.902667    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 04:48:58.902824 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.902677    2573 projected.go:196] Error preparing data for projected volume kube-api-access-hsv87 for pod openshift-network-diagnostics/network-check-target-64c4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:48:58.902824 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:58.902747    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87 podName:c0c4f544-8e7a-4ffe-bd00-96cc6057edc5 nodeName:}" failed. No retries permitted until 2026-09-16 04:48:59.402729057 +0000 UTC m=+3.068658630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv87" (UniqueName: "kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87") pod "network-check-target-64c4f" (UID: "c0c4f544-8e7a-4ffe-bd00-96cc6057edc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:48:58.904969 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.904946    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f784j\" (UniqueName: \"kubernetes.io/projected/b33becd7-76a1-4999-93f7-94222c51c7fd-kube-api-access-f784j\") pod \"tuned-5cpkt\" (UID: \"b33becd7-76a1-4999-93f7-94222c51c7fd\") " pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:58.905884 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.905866    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mcl2\" (UniqueName: \"kubernetes.io/projected/7b25e368-49cf-4c2e-840d-2347f6ceb400-kube-api-access-7mcl2\") pod \"multus-78g6m\" (UID: \"7b25e368-49cf-4c2e-840d-2347f6ceb400\") " pod="openshift-multus/multus-78g6m"
Sep 16 04:48:58.906643 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.906620    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlglz\" (UniqueName: \"kubernetes.io/projected/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-kube-api-access-jlglz\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:48:58.907140 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.907123    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq2mc\" (UniqueName: \"kubernetes.io/projected/9c097b7c-cb2b-4fec-844f-72d0f78a12fe-kube-api-access-kq2mc\") pod \"iptables-alerter-zrwk4\" (UID: \"9c097b7c-cb2b-4fec-844f-72d0f78a12fe\") " pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:58.987263 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.987202    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2tvkn"
Sep 16 04:48:58.995748 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:58.995729    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-69szj"
Sep 16 04:48:59.004677 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.004651    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw"
Sep 16 04:48:59.010933 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.010913    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rw649"
Sep 16 04:48:59.017848 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.017827    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:48:59.024738 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.024714    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-78g6m"
Sep 16 04:48:59.032751 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.032731    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zrwk4"
Sep 16 04:48:59.041528 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.041507    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5cpkt"
Sep 16 04:48:59.044871 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.044847    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:48:59.135518 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.135492    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 04:48:59.400357 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.400278    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:48:59.400357 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.400336    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:48:59.400562 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:59.400402    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 04:48:59.400562 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:59.400425    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:48:59.400562 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:59.400475    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret podName:b9705e01-c8d0-481f-8769-ffd767bc202c nodeName:}" failed. No retries permitted until 2026-09-16 04:49:00.400457463 +0000 UTC m=+4.066387054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret") pod "global-pull-secret-syncer-vqmw5" (UID: "b9705e01-c8d0-481f-8769-ffd767bc202c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 04:48:59.400562 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:59.400492    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs podName:835f4ee4-d6b0-4fb6-b6e6-86277d856c67 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:00.400484929 +0000 UTC m=+4.066414501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs") pod "network-metrics-daemon-k449w" (UID: "835f4ee4-d6b0-4fb6-b6e6-86277d856c67") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:48:59.501488 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.501454    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv87\" (UniqueName: \"kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87\") pod \"network-check-target-64c4f\" (UID: \"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5\") " pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:48:59.501666 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:59.501614    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 04:48:59.501666 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:59.501647    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 04:48:59.501666 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:59.501661    2573 projected.go:196] Error preparing data for projected volume kube-api-access-hsv87 for pod openshift-network-diagnostics/network-check-target-64c4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:48:59.501810 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:59.501725    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87 podName:c0c4f544-8e7a-4ffe-bd00-96cc6057edc5 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:00.501704132 +0000 UTC m=+4.167633721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv87" (UniqueName: "kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87") pod "network-check-target-64c4f" (UID: "c0c4f544-8e7a-4ffe-bd00-96cc6057edc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:48:59.727285 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.727175    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 04:43:59 +0000 UTC" deadline="2028-02-25 13:46:15.842086212 +0000 UTC"
Sep 16 04:48:59.727285 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.727239    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12656h57m16.114850372s"
Sep 16 04:48:59.746154 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:59.746124    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b25e368_49cf_4c2e_840d_2347f6ceb400.slice/crio-c6350f7d837508d0eb79cf705c74b9a553344c08bb7b3da78b76884bb596e5c7 WatchSource:0}: Error finding container c6350f7d837508d0eb79cf705c74b9a553344c08bb7b3da78b76884bb596e5c7: Status 404 returned error can't find the container with id c6350f7d837508d0eb79cf705c74b9a553344c08bb7b3da78b76884bb596e5c7
Sep 16 04:48:59.750305 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:59.750276    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacab9663_a074_4fbb_8596_a50e5e39b878.slice/crio-5b212b32317299c6d86320f2e34f8d1317fce299144e3ab7c6de30cbb232a5d7 WatchSource:0}: Error finding container 5b212b32317299c6d86320f2e34f8d1317fce299144e3ab7c6de30cbb232a5d7: Status 404 returned error can't find the container with id 5b212b32317299c6d86320f2e34f8d1317fce299144e3ab7c6de30cbb232a5d7
Sep 16 04:48:59.753465 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:59.753436    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d65b077_2a03_48e7_ba99_7bd68b698d75.slice/crio-b9eec450a0ddffd878c06bfef532893cd3a45c7801f69f867a5f6a0e32653c1b WatchSource:0}: Error finding container b9eec450a0ddffd878c06bfef532893cd3a45c7801f69f867a5f6a0e32653c1b: Status 404 returned error can't find the container with id b9eec450a0ddffd878c06bfef532893cd3a45c7801f69f867a5f6a0e32653c1b
Sep 16 04:48:59.755701 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:59.755660    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6985599b_f750_48f3_9e34_55dd74ed94bc.slice/crio-fe9c71e45bff194ad5c236771c611c761bbec9a15add792c3d0443860b5bc6db WatchSource:0}: Error finding container fe9c71e45bff194ad5c236771c611c761bbec9a15add792c3d0443860b5bc6db: Status 404 returned error can't find the container with id fe9c71e45bff194ad5c236771c611c761bbec9a15add792c3d0443860b5bc6db
Sep 16 04:48:59.756561 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:59.756499    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c097b7c_cb2b_4fec_844f_72d0f78a12fe.slice/crio-796f69a3bca552d61893b66f1e5ed6e6001c0c586b79bb1c600ca2ffff7accbc WatchSource:0}: Error finding container 796f69a3bca552d61893b66f1e5ed6e6001c0c586b79bb1c600ca2ffff7accbc: Status 404 returned error can't find the container with id 796f69a3bca552d61893b66f1e5ed6e6001c0c586b79bb1c600ca2ffff7accbc
Sep 16 04:48:59.779058 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:59.779027    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b6aa0b_369b_4585_bcbc_890512014439.slice/crio-10e5711bec16026be35b8e8a89bca65fa394af89a9c67f1fbd5c07c7072d2a95 WatchSource:0}: Error finding container 10e5711bec16026be35b8e8a89bca65fa394af89a9c67f1fbd5c07c7072d2a95: Status 404 returned error can't find the container with id 10e5711bec16026be35b8e8a89bca65fa394af89a9c67f1fbd5c07c7072d2a95
Sep 16 04:48:59.779769 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.779747    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:48:59.779769 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:59.779761    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64223cf5_19e6_4f4a_aa87_67766e0373d9.slice/crio-0f5bac492e72f6eee313e60a141c7f5ccf581ece730a58cfb2103472b2920ade WatchSource:0}: Error finding container 0f5bac492e72f6eee313e60a141c7f5ccf581ece730a58cfb2103472b2920ade: Status 404 returned error can't find the container with id 0f5bac492e72f6eee313e60a141c7f5ccf581ece730a58cfb2103472b2920ade
Sep 16 04:48:59.779915 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:48:59.779859    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:48:59.780401 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:59.780377    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33becd7_76a1_4999_93f7_94222c51c7fd.slice/crio-7200907dce1d961af06d3b47f2bd99b9cdf90ae38c4a72a90054c35b73322efe WatchSource:0}: Error finding container 7200907dce1d961af06d3b47f2bd99b9cdf90ae38c4a72a90054c35b73322efe: Status 404 returned error can't find the container with id 7200907dce1d961af06d3b47f2bd99b9cdf90ae38c4a72a90054c35b73322efe
Sep 16 04:48:59.781457 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:48:59.781405    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a9d2ea_0102_4624_bfa6_b96860b1f483.slice/crio-9c303777bb8bee9dd482ac91de080faa9c4084c88426b003738ea55913743c2d WatchSource:0}: Error finding container 9c303777bb8bee9dd482ac91de080faa9c4084c88426b003738ea55913743c2d: Status 404 returned error can't find the container with id 9c303777bb8bee9dd482ac91de080faa9c4084c88426b003738ea55913743c2d
Sep 16 04:48:59.790232 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.790207    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-78g6m" event={"ID":"7b25e368-49cf-4c2e-840d-2347f6ceb400","Type":"ContainerStarted","Data":"c6350f7d837508d0eb79cf705c74b9a553344c08bb7b3da78b76884bb596e5c7"}
Sep 16 04:48:59.791742 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.791716    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-24mv8" event={"ID":"36a9d2ea-0102-4624-bfa6-b96860b1f483","Type":"ContainerStarted","Data":"9c303777bb8bee9dd482ac91de080faa9c4084c88426b003738ea55913743c2d"}
Sep 16 04:48:59.792978 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.792934    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5cpkt" event={"ID":"b33becd7-76a1-4999-93f7-94222c51c7fd","Type":"ContainerStarted","Data":"7200907dce1d961af06d3b47f2bd99b9cdf90ae38c4a72a90054c35b73322efe"}
Sep 16 04:48:59.794416 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.794387    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tvkn" event={"ID":"acab9663-a074-4fbb-8596-a50e5e39b878","Type":"ContainerStarted","Data":"5b212b32317299c6d86320f2e34f8d1317fce299144e3ab7c6de30cbb232a5d7"}
Sep 16 04:48:59.795808 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.795771    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-69szj" event={"ID":"64223cf5-19e6-4f4a-aa87-67766e0373d9","Type":"ContainerStarted","Data":"0f5bac492e72f6eee313e60a141c7f5ccf581ece730a58cfb2103472b2920ade"}
Sep 16 04:48:59.797141 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.797114    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rw649" event={"ID":"01b6aa0b-369b-4585-bcbc-890512014439","Type":"ContainerStarted","Data":"10e5711bec16026be35b8e8a89bca65fa394af89a9c67f1fbd5c07c7072d2a95"}
Sep 16 04:48:59.798435 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.798409    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zrwk4" event={"ID":"9c097b7c-cb2b-4fec-844f-72d0f78a12fe","Type":"ContainerStarted","Data":"796f69a3bca552d61893b66f1e5ed6e6001c0c586b79bb1c600ca2ffff7accbc"}
Sep 16 04:48:59.799741 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.799716    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" event={"ID":"6985599b-f750-48f3-9e34-55dd74ed94bc","Type":"ContainerStarted","Data":"fe9c71e45bff194ad5c236771c611c761bbec9a15add792c3d0443860b5bc6db"}
Sep 16 04:48:59.801013 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:48:59.800991    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw" event={"ID":"9d65b077-2a03-48e7-ba99-7bd68b698d75","Type":"ContainerStarted","Data":"b9eec450a0ddffd878c06bfef532893cd3a45c7801f69f867a5f6a0e32653c1b"}
Sep 16 04:49:00.409120 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.408268    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:00.409120 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.408327    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:00.409120 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:00.408479    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:49:00.409120 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:00.408555    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs podName:835f4ee4-d6b0-4fb6-b6e6-86277d856c67 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:02.408523696 +0000 UTC m=+6.074453284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs") pod "network-metrics-daemon-k449w" (UID: "835f4ee4-d6b0-4fb6-b6e6-86277d856c67") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:49:00.409120 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:00.408988    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 04:49:00.409120 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:00.409041    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret podName:b9705e01-c8d0-481f-8769-ffd767bc202c nodeName:}" failed. No retries permitted until 2026-09-16 04:49:02.409026213 +0000 UTC m=+6.074955791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret") pod "global-pull-secret-syncer-vqmw5" (UID: "b9705e01-c8d0-481f-8769-ffd767bc202c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 04:49:00.509288 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.509249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv87\" (UniqueName: \"kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87\") pod \"network-check-target-64c4f\" (UID: \"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5\") " pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:00.509495 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:00.509475    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 04:49:00.509568 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:00.509500    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 04:49:00.509568 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:00.509514    2573 projected.go:196] Error preparing data for projected volume kube-api-access-hsv87 for pod openshift-network-diagnostics/network-check-target-64c4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:49:00.509708 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:00.509570    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87 podName:c0c4f544-8e7a-4ffe-bd00-96cc6057edc5 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:02.509552978 +0000 UTC m=+6.175482557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv87" (UniqueName: "kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87") pod "network-check-target-64c4f" (UID: "c0c4f544-8e7a-4ffe-bd00-96cc6057edc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:49:00.728198 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.728111    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 04:43:59 +0000 UTC" deadline="2028-05-05 20:14:37.23919278 +0000 UTC"
Sep 16 04:49:00.728198 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.728148    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14343h25m36.511049684s"
Sep 16 04:49:00.786847 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.786817    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:00.787014 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:00.786981    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:00.787446 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.787423    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:00.787557 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:00.787527    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:00.837632 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.837598    2573 generic.go:356] "Generic (PLEG): container finished" podID="4d4e64cb672ced1a149e7f33c599dbd9" containerID="223c56243d85740f04602f13fe37b12be557c726fdf276588d879d1ae9e5e2e5" exitCode=0
Sep 16 04:49:00.837798 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.837704    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal" event={"ID":"4d4e64cb672ced1a149e7f33c599dbd9","Type":"ContainerDied","Data":"223c56243d85740f04602f13fe37b12be557c726fdf276588d879d1ae9e5e2e5"}
Sep 16 04:49:00.873501 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.873468    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal" event={"ID":"461b5598d7024278d7832d79de8a2c73","Type":"ContainerStarted","Data":"16a2f4628df2b5a346934a01f43250a7605f015043cf8c04329804ab7cc868f5"}
Sep 16 04:49:00.891224 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:00.890556    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-117.ec2.internal" podStartSLOduration=2.890539391 podStartE2EDuration="2.890539391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:00.889961689 +0000 UTC m=+4.555891290" watchObservedRunningTime="2026-09-16 04:49:00.890539391 +0000 UTC m=+4.556468988"
Sep 16 04:49:01.781253 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:01.780826    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:01.781253 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:01.781212    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:01.880421 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:01.880336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal" event={"ID":"4d4e64cb672ced1a149e7f33c599dbd9","Type":"ContainerStarted","Data":"8d4fde226d1b0df74dc1a57da3a6b6fb750b726f4e9fe8503162110b4ae0bfad"}
Sep 16 04:49:02.428032 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:02.427988    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:02.428238 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:02.428049    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:02.428238 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:02.428205    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:49:02.428359 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:02.428268    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs podName:835f4ee4-d6b0-4fb6-b6e6-86277d856c67 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:06.428249631 +0000 UTC m=+10.094179217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs") pod "network-metrics-daemon-k449w" (UID: "835f4ee4-d6b0-4fb6-b6e6-86277d856c67") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:49:02.428705 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:02.428684    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 04:49:02.428795 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:02.428739    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret podName:b9705e01-c8d0-481f-8769-ffd767bc202c nodeName:}" failed. No retries permitted until 2026-09-16 04:49:06.428724048 +0000 UTC m=+10.094653635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret") pod "global-pull-secret-syncer-vqmw5" (UID: "b9705e01-c8d0-481f-8769-ffd767bc202c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 04:49:02.529065 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:02.529029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv87\" (UniqueName: \"kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87\") pod \"network-check-target-64c4f\" (UID: \"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5\") " pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:02.529247 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:02.529193    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 04:49:02.529247 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:02.529217    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 04:49:02.529247 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:02.529231    2573 projected.go:196] Error preparing data for projected volume kube-api-access-hsv87 for pod openshift-network-diagnostics/network-check-target-64c4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:49:02.529403 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:02.529283    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87 podName:c0c4f544-8e7a-4ffe-bd00-96cc6057edc5 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:06.529265597 +0000 UTC m=+10.195195184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv87" (UniqueName: "kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87") pod "network-check-target-64c4f" (UID: "c0c4f544-8e7a-4ffe-bd00-96cc6057edc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:49:02.781935 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:02.781783    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:02.781935 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:02.781921    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:02.782457 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:02.781977    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:02.782457 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:02.782105    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:03.780692 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:03.780643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:03.780886 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:03.780774    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:04.782702 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:04.781717    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:04.782702 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:04.781866    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:04.782702 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:04.782388    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:04.782702 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:04.782509    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:05.780394 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:05.780320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:05.780559 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:05.780503    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:06.463000 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:06.462962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:06.463432 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:06.463014    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:06.463432 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:06.463117    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 04:49:06.463432 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:06.463168    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:49:06.463432 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:06.463189    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret podName:b9705e01-c8d0-481f-8769-ffd767bc202c nodeName:}" failed. No retries permitted until 2026-09-16 04:49:14.463168355 +0000 UTC m=+18.129097939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret") pod "global-pull-secret-syncer-vqmw5" (UID: "b9705e01-c8d0-481f-8769-ffd767bc202c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 04:49:06.463432 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:06.463217    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs podName:835f4ee4-d6b0-4fb6-b6e6-86277d856c67 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:14.463200685 +0000 UTC m=+18.129130270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs") pod "network-metrics-daemon-k449w" (UID: "835f4ee4-d6b0-4fb6-b6e6-86277d856c67") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:49:06.564229 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:06.564195    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv87\" (UniqueName: \"kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87\") pod \"network-check-target-64c4f\" (UID: \"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5\") " pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:06.564387 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:06.564329    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 04:49:06.564387 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:06.564355    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 04:49:06.564387 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:06.564369    2573 projected.go:196] Error preparing data for projected volume kube-api-access-hsv87 for pod openshift-network-diagnostics/network-check-target-64c4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:49:06.564508 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:06.564428    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87 podName:c0c4f544-8e7a-4ffe-bd00-96cc6057edc5 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:14.564410925 +0000 UTC m=+18.230340509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv87" (UniqueName: "kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87") pod "network-check-target-64c4f" (UID: "c0c4f544-8e7a-4ffe-bd00-96cc6057edc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:49:06.780814 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:06.780748    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:06.780950 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:06.780850    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:06.781340 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:06.781197    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:06.781340 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:06.781299    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:07.780500 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:07.780470    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:07.780956 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:07.780578    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:08.781009 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:08.780109    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:08.781009 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:08.780256    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:08.781009 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:08.780764    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:08.781009 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:08.780859    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:09.780487 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:09.780453    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:09.780650 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:09.780554    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:10.780325 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:10.780298    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:10.780726 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:10.780311    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:10.780726 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:10.780436    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:10.780726 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:10.780535    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:11.779734 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:11.779702    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:11.779896 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:11.779801    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:12.779826 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:12.779787    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:12.780270 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:12.779918    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:12.780270 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:12.780008    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:12.780270 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:12.780115    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:13.780641 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:13.780609    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:13.781071 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:13.780714    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:14.530002 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:14.529966    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:14.530002 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:14.530007    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:14.530237 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:14.530114    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:49:14.530237 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:14.530117    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 04:49:14.530237 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:14.530178    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs podName:835f4ee4-d6b0-4fb6-b6e6-86277d856c67 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:30.530158534 +0000 UTC m=+34.196088108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs") pod "network-metrics-daemon-k449w" (UID: "835f4ee4-d6b0-4fb6-b6e6-86277d856c67") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:49:14.530237 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:14.530193    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret podName:b9705e01-c8d0-481f-8769-ffd767bc202c nodeName:}" failed. No retries permitted until 2026-09-16 04:49:30.530187074 +0000 UTC m=+34.196116644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret") pod "global-pull-secret-syncer-vqmw5" (UID: "b9705e01-c8d0-481f-8769-ffd767bc202c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 04:49:14.630541 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:14.630474    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv87\" (UniqueName: \"kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87\") pod \"network-check-target-64c4f\" (UID: \"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5\") " pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:14.630721 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:14.630631    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 04:49:14.630721 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:14.630653    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 04:49:14.630721 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:14.630666    2573 projected.go:196] Error preparing data for projected volume kube-api-access-hsv87 for pod openshift-network-diagnostics/network-check-target-64c4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:49:14.630878 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:14.630726    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87 podName:c0c4f544-8e7a-4ffe-bd00-96cc6057edc5 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:30.630708606 +0000 UTC m=+34.296638196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv87" (UniqueName: "kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87") pod "network-check-target-64c4f" (UID: "c0c4f544-8e7a-4ffe-bd00-96cc6057edc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:49:14.780370 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:14.780287    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:14.780516 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:14.780381    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:14.780516 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:14.780400    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:14.780516 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:14.780497    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:15.780221 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:15.780189    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:15.780648 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:15.780312    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:16.781092 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:16.781053    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:16.781526 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:16.781153    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:16.781526 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:16.781227    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:16.781526 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:16.781312    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:17.779891 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:17.779867    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:17.780001 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:17.779961    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:18.781229 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.780829    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:18.781229 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.780880    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:18.782030 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:18.781256    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:18.782030 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:18.781309    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:18.916978 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.916950    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw" event={"ID":"9d65b077-2a03-48e7-ba99-7bd68b698d75","Type":"ContainerStarted","Data":"65a204bc1f9a0c4c2cd9997c5858e129781bc05105310ef718ba3eea7a8fb038"}
Sep 16 04:49:18.918273 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.918254    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-78g6m" event={"ID":"7b25e368-49cf-4c2e-840d-2347f6ceb400","Type":"ContainerStarted","Data":"a3229b6f60eebfaf1c26b700271487eea7e4bcc84bbde961233bfdb37735b7b7"}
Sep 16 04:49:18.919558 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.919534    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-24mv8" event={"ID":"36a9d2ea-0102-4624-bfa6-b96860b1f483","Type":"ContainerStarted","Data":"873b65f05e245fbfa0ad0fbd805ad4bd8fbc31fb6b569d485d7107ba11cee27f"}
Sep 16 04:49:18.920866 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.920844    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5cpkt" event={"ID":"b33becd7-76a1-4999-93f7-94222c51c7fd","Type":"ContainerStarted","Data":"b5f31b468bd8ea2563be5c982235a2e2e90f1d57772595789e052200b6f79f4c"}
Sep 16 04:49:18.922171 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.922153    2573 generic.go:356] "Generic (PLEG): container finished" podID="acab9663-a074-4fbb-8596-a50e5e39b878" containerID="54f53f02f0a53ae12f0b6e3944abc422777730aea82744f3ce50f3d8feb37e62" exitCode=0
Sep 16 04:49:18.922241 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.922211    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tvkn" event={"ID":"acab9663-a074-4fbb-8596-a50e5e39b878","Type":"ContainerDied","Data":"54f53f02f0a53ae12f0b6e3944abc422777730aea82744f3ce50f3d8feb37e62"}
Sep 16 04:49:18.923638 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.923618    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-69szj" event={"ID":"64223cf5-19e6-4f4a-aa87-67766e0373d9","Type":"ContainerStarted","Data":"f25770e7944df3f546ab376e28c840ba0df208ebfcc7b5a385110305b7c4b074"}
Sep 16 04:49:18.924975 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.924950    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rw649" event={"ID":"01b6aa0b-369b-4585-bcbc-890512014439","Type":"ContainerStarted","Data":"baa6fbcbcdc85a4622695f53b0eca7e40283c46b606300b9e4b3cafdce4b6b35"}
Sep 16 04:49:18.927346 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.927324    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 04:49:18.927729 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.927715    2573 generic.go:356] "Generic (PLEG): container finished" podID="6985599b-f750-48f3-9e34-55dd74ed94bc" containerID="e38be231af543ea4e8371701766ccabdb38621ee0c88a77423d2bceeb407b903" exitCode=1
Sep 16 04:49:18.927791 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.927737    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" event={"ID":"6985599b-f750-48f3-9e34-55dd74ed94bc","Type":"ContainerStarted","Data":"b3d12a60a5129b0188c4e357a2fdccfe7ca04e13aa3f064ef701e2c636eb6643"}
Sep 16 04:49:18.927791 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.927782    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" event={"ID":"6985599b-f750-48f3-9e34-55dd74ed94bc","Type":"ContainerStarted","Data":"320a1d6d1f494b6c5dbc39587d9d10cf2ae5d806058959e1fbdc930b5805fde4"}
Sep 16 04:49:18.927850 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.927811    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" event={"ID":"6985599b-f750-48f3-9e34-55dd74ed94bc","Type":"ContainerStarted","Data":"cf06110291247a0d3790259dbb735660b376760153cd044cc4577098b408ed50"}
Sep 16 04:49:18.927850 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.927838    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" event={"ID":"6985599b-f750-48f3-9e34-55dd74ed94bc","Type":"ContainerStarted","Data":"bc661fe20124e078afb04d5501ef9aa8c367b5fa48c774cd27b52a320f10c0a0"}
Sep 16 04:49:18.927910 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.927863    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" event={"ID":"6985599b-f750-48f3-9e34-55dd74ed94bc","Type":"ContainerDied","Data":"e38be231af543ea4e8371701766ccabdb38621ee0c88a77423d2bceeb407b903"}
Sep 16 04:49:18.927910 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.927891    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" event={"ID":"6985599b-f750-48f3-9e34-55dd74ed94bc","Type":"ContainerStarted","Data":"35474f0206926e398c3034a57478acf9a3358ab65cd557be017c1b6f99516f5d"}
Sep 16 04:49:18.932668 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.932630    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-78g6m" podStartSLOduration=3.635869943 podStartE2EDuration="21.932583543s" totalImagesPullingTime="18.2967136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:18.93255336 +0000 UTC m=+22.598482974" watchObservedRunningTime="2026-09-16 04:49:18.932583543 +0000 UTC m=+22.598513158"
Sep 16 04:49:18.933270 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.933244    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-117.ec2.internal" podStartSLOduration=20.933234488 podStartE2EDuration="20.933234488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:01.898935349 +0000 UTC m=+5.564864960" watchObservedRunningTime="2026-09-16 04:49:18.933234488 +0000 UTC m=+22.599164080"
Sep 16 04:49:18.945057 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.945025    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-69szj" podStartSLOduration=3.867362121 podStartE2EDuration="21.945014144s" totalImagesPullingTime="18.077652023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:18.944989846 +0000 UTC m=+22.610919456" watchObservedRunningTime="2026-09-16 04:49:18.945014144 +0000 UTC m=+22.610943736"
Sep 16 04:49:18.957307 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.957272    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-24mv8" podStartSLOduration=11.248809779 podStartE2EDuration="21.957262525s" totalImagesPullingTime="10.708452746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:18.956900849 +0000 UTC m=+22.622830462" watchObservedRunningTime="2026-09-16 04:49:18.957262525 +0000 UTC m=+22.623192116"
Sep 16 04:49:18.970864 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.970830    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rw649" podStartSLOduration=3.88986994 podStartE2EDuration="21.970819792s" totalImagesPullingTime="18.080949852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:18.970478008 +0000 UTC m=+22.636407618" watchObservedRunningTime="2026-09-16 04:49:18.970819792 +0000 UTC m=+22.636749383"
Sep 16 04:49:18.984224 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:18.984192    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5cpkt" podStartSLOduration=3.903591205 podStartE2EDuration="21.98418362s" totalImagesPullingTime="18.080592415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:18.983864872 +0000 UTC m=+22.649794619" watchObservedRunningTime="2026-09-16 04:49:18.98418362 +0000 UTC m=+22.650113211"
Sep 16 04:49:19.339608 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:19.339578    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 04:49:19.743954 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:19.743859    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T04:49:19.339592419Z","UUID":"4dc29b7b-775c-4c83-a28d-840d6027f53d","Handler":null,"Name":"","Endpoint":""}
Sep 16 04:49:19.745462 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:19.745444    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 04:49:19.745588 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:19.745474    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 04:49:19.779973 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:19.779947    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:19.780097 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:19.780061    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:19.935220 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:19.935189    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw" event={"ID":"9d65b077-2a03-48e7-ba99-7bd68b698d75","Type":"ContainerStarted","Data":"e33a42615107a248bea6bef675bea1586d23d24b3f718d5cea386e0ed2ff9864"}
Sep 16 04:49:19.936832 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:19.936805    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zrwk4" event={"ID":"9c097b7c-cb2b-4fec-844f-72d0f78a12fe","Type":"ContainerStarted","Data":"8270d7c07a58ab7776a6f07fb271d9da28adf30ae094f22d19badbfa0bd54018"}
Sep 16 04:49:19.952981 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:19.952937    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zrwk4" podStartSLOduration=4.869020717 podStartE2EDuration="22.952921574s" totalImagesPullingTime="18.083900857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:19.952645386 +0000 UTC m=+23.618575005" watchObservedRunningTime="2026-09-16 04:49:19.952921574 +0000 UTC m=+23.618851168"
Sep 16 04:49:20.780045 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:20.780022    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:20.780189 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:20.780134    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:20.780264 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:20.780210    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:20.780349 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:20.780321    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:20.795842 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:20.795820    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:49:20.942877 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:20.942822    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 04:49:20.943275 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:20.943255    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" event={"ID":"6985599b-f750-48f3-9e34-55dd74ed94bc","Type":"ContainerStarted","Data":"e667787c2903b60fa38b49f27cb23114b8e93b25fac674566dc0ef06c4cde161"}
Sep 16 04:49:20.945432 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:20.945408    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw" event={"ID":"9d65b077-2a03-48e7-ba99-7bd68b698d75","Type":"ContainerStarted","Data":"4c0105f688f3465b8b9ec3ed5d4f06f05b1a7d2c3f13ec86519c9b539a108a20"}
Sep 16 04:49:20.962153 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:20.962114    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6lhw" podStartSLOduration=3.797817308 podStartE2EDuration="23.962101974s" totalImagesPullingTime="20.164284666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:20.961408074 +0000 UTC m=+24.627337686" watchObservedRunningTime="2026-09-16 04:49:20.962101974 +0000 UTC m=+24.628031566"
Sep 16 04:49:21.780538 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:21.780508    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:21.780720 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:21.780635    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:22.781320 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:22.781165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:22.781948 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:22.781735    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:22.781948 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:22.781754    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:22.781948 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:22.781895    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:22.954294 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:22.954274    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 04:49:22.954717 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:22.954693    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" event={"ID":"6985599b-f750-48f3-9e34-55dd74ed94bc","Type":"ContainerStarted","Data":"f3914ddf2274f89aac78d4db47134753383eb322ebe3380beb6e33dee6d81715"}
Sep 16 04:49:22.954932 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:22.954909    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:49:22.955103 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:22.955088    2573 scope.go:117] "RemoveContainer" containerID="e38be231af543ea4e8371701766ccabdb38621ee0c88a77423d2bceeb407b903"
Sep 16 04:49:22.956748 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:22.956728    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tvkn" event={"ID":"acab9663-a074-4fbb-8596-a50e5e39b878","Type":"ContainerStarted","Data":"34b86230cc9295355c1e1993da84d3c7127f05e5b88b31bf132b063fd765cfa1"}
Sep 16 04:49:22.970629 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:22.970585    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:49:23.385876 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.385820    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:49:23.386492 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.386340    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:49:23.780384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.780363    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:23.780479 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:23.780441    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:23.962958 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.962934    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 04:49:23.963622 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.963366    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" event={"ID":"6985599b-f750-48f3-9e34-55dd74ed94bc","Type":"ContainerStarted","Data":"86248787b61ce8e94043f4f69e146c79dc926fb8f70d748788303f42327f1966"}
Sep 16 04:49:23.963695 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.963648    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:49:23.963744 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.963722    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:49:23.965506 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.965479    2573 generic.go:356] "Generic (PLEG): container finished" podID="acab9663-a074-4fbb-8596-a50e5e39b878" containerID="34b86230cc9295355c1e1993da84d3c7127f05e5b88b31bf132b063fd765cfa1" exitCode=0
Sep 16 04:49:23.965627 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.965570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tvkn" event={"ID":"acab9663-a074-4fbb-8596-a50e5e39b878","Type":"ContainerDied","Data":"34b86230cc9295355c1e1993da84d3c7127f05e5b88b31bf132b063fd765cfa1"}
Sep 16 04:49:23.966298 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.966272    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-24mv8"
Sep 16 04:49:23.980383 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.980366    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:49:23.988290 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:23.988253    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88" podStartSLOduration=8.871273297 podStartE2EDuration="26.988242504s" totalImagesPullingTime="18.116969207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:23.987977957 +0000 UTC m=+27.653907641" watchObservedRunningTime="2026-09-16 04:49:23.988242504 +0000 UTC m=+27.654172078"
Sep 16 04:49:24.754743 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:24.754715    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k449w"]
Sep 16 04:49:24.754982 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:24.754836    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:24.754982 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:24.754958    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:24.755467 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:24.755441    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vqmw5"]
Sep 16 04:49:24.755559 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:24.755547    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:24.755681 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:24.755660    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:24.765010 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:24.764957    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-64c4f"]
Sep 16 04:49:24.765075 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:24.765055    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:24.765163 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:24.765143    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:25.780270 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:25.780208    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:25.780568 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:25.780320    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:25.972478 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:25.972450    2573 generic.go:356] "Generic (PLEG): container finished" podID="acab9663-a074-4fbb-8596-a50e5e39b878" containerID="c99efa6a8aa8b7311d0219f45b74c8bb8a05a1b58617e50a4cfcb5e6e3a9be99" exitCode=0
Sep 16 04:49:25.972618 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:25.972534    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tvkn" event={"ID":"acab9663-a074-4fbb-8596-a50e5e39b878","Type":"ContainerDied","Data":"c99efa6a8aa8b7311d0219f45b74c8bb8a05a1b58617e50a4cfcb5e6e3a9be99"}
Sep 16 04:49:26.781017 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:26.780986    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:26.781363 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:26.781063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:26.781363 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:26.781093    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:26.781363 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:26.781116    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:26.977336 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:26.977118    2573 generic.go:356] "Generic (PLEG): container finished" podID="acab9663-a074-4fbb-8596-a50e5e39b878" containerID="d09556b4789beb468667e3cd74fa732a08dc363e4fd0f13420ca3f515dc5a9c2" exitCode=0
Sep 16 04:49:26.977336 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:26.977192    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tvkn" event={"ID":"acab9663-a074-4fbb-8596-a50e5e39b878","Type":"ContainerDied","Data":"d09556b4789beb468667e3cd74fa732a08dc363e4fd0f13420ca3f515dc5a9c2"}
Sep 16 04:49:27.780010 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:27.779933    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:27.780163 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:27.780050    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:28.780104 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:28.780078    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:28.780554 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:28.780192    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqmw5" podUID="b9705e01-c8d0-481f-8769-ffd767bc202c"
Sep 16 04:49:28.780554 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:28.780261    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:28.780554 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:28.780348    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-64c4f" podUID="c0c4f544-8e7a-4ffe-bd00-96cc6057edc5"
Sep 16 04:49:28.986996 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:28.986935    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:29.097341 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:29.097258    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:29.200750 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:29.200719    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:29.305867 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:29.305832    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:29.780697 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:29.780670    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:29.781427 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:29.780809    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k449w" podUID="835f4ee4-d6b0-4fb6-b6e6-86277d856c67"
Sep 16 04:49:30.199522 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.199443    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:30.300379 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.300344    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:30.555332 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.555302    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:30.555469 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.555353    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:30.555469 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.555457    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 04:49:30.555581 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.555520    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret podName:b9705e01-c8d0-481f-8769-ffd767bc202c nodeName:}" failed. No retries permitted until 2026-09-16 04:50:02.55550054 +0000 UTC m=+66.221430123 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret") pod "global-pull-secret-syncer-vqmw5" (UID: "b9705e01-c8d0-481f-8769-ffd767bc202c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 04:49:30.555581 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.555457    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:49:30.555717 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.555633    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs podName:835f4ee4-d6b0-4fb6-b6e6-86277d856c67 nodeName:}" failed. No retries permitted until 2026-09-16 04:50:02.555614067 +0000 UTC m=+66.221543654 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs") pod "network-metrics-daemon-k449w" (UID: "835f4ee4-d6b0-4fb6-b6e6-86277d856c67") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 04:49:30.601335 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.601313    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-117.ec2.internal" event="NodeReady"
Sep 16 04:49:30.601473 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.601455    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 04:49:30.634923 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.634899    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-675cb69c4f-hx57l"]
Sep 16 04:49:30.656203 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.656179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv87\" (UniqueName: \"kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87\") pod \"network-check-target-64c4f\" (UID: \"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5\") " pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:30.656376 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.656357    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 04:49:30.656475 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.656383    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 04:49:30.656475 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.656397    2573 projected.go:196] Error preparing data for projected volume kube-api-access-hsv87 for pod openshift-network-diagnostics/network-check-target-64c4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:49:30.656475 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.656451    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87 podName:c0c4f544-8e7a-4ffe-bd00-96cc6057edc5 nodeName:}" failed. No retries permitted until 2026-09-16 04:50:02.656434717 +0000 UTC m=+66.322364309 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv87" (UniqueName: "kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87") pod "network-check-target-64c4f" (UID: "c0c4f544-8e7a-4ffe-bd00-96cc6057edc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 04:49:30.677897 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.677874    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4jtn2"]
Sep 16 04:49:30.678048 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.678017    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.680026 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.680004    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 04:49:30.680130 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.680039    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tff4l\""
Sep 16 04:49:30.680188 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.680126    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 04:49:30.680536 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.680514    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 04:49:30.687644 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.687623    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 04:49:30.707274 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.707229    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hfn7r"]
Sep 16 04:49:30.707387 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.707369    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4jtn2"
Sep 16 04:49:30.709493 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.709359    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 04:49:30.709493 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.709458    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 04:49:30.709493 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.709457    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cl5jq\""
Sep 16 04:49:30.709493 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.709488    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 04:49:30.725486 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.725466    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675cb69c4f-hx57l"]
Sep 16 04:49:30.725569 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.725497    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4jtn2"]
Sep 16 04:49:30.725569 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.725516    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hfn7r"]
Sep 16 04:49:30.725692 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.725619    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.727525 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.727509    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 04:49:30.727634 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.727551    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 04:49:30.727634 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.727621    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgbjt\""
Sep 16 04:49:30.774866 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.774841    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:30.780773 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.780748    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:49:30.781213 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.780755    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:49:30.783007 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.782987    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 04:49:30.783098 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.783026    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 04:49:30.783252 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.783235    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 04:49:30.783313 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.783288    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gbd6v\""
Sep 16 04:49:30.835935 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.835906    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:30.857231 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-trusted-ca\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.857317 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857233    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-bound-sa-token\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.857317 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-image-registry-private-configuration\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.857317 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.857439 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxv2\" (UniqueName: \"kubernetes.io/projected/5d19721b-2447-4d1f-b913-df5646e2baa0-kube-api-access-szxv2\") pod \"ingress-canary-4jtn2\" (UID: \"5d19721b-2447-4d1f-b913-df5646e2baa0\") " pod="openshift-ingress-canary/ingress-canary-4jtn2"
Sep 16 04:49:30.857439 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/102f3bea-c75e-4dea-9234-83c4785a68fc-ca-trust-extracted\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.857439 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857388    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnntg\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-kube-api-access-fnntg\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.857439 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857411    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hpp\" (UniqueName: \"kubernetes.io/projected/58152a4a-74ba-4f56-a870-99a5bae3e7af-kube-api-access-s6hpp\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.857439 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d19721b-2447-4d1f-b913-df5646e2baa0-cert\") pod \"ingress-canary-4jtn2\" (UID: \"5d19721b-2447-4d1f-b913-df5646e2baa0\") " pod="openshift-ingress-canary/ingress-canary-4jtn2"
Sep 16 04:49:30.857631 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58152a4a-74ba-4f56-a870-99a5bae3e7af-tmp-dir\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.857631 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.857631 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857574    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-certificates\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.857631 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58152a4a-74ba-4f56-a870-99a5bae3e7af-config-volume\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.857631 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.857625    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-installation-pull-secrets\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.958778 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.958735    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58152a4a-74ba-4f56-a870-99a5bae3e7af-tmp-dir\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.958778 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.958770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.958881 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.958787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-certificates\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.958881 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.958829    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58152a4a-74ba-4f56-a870-99a5bae3e7af-config-volume\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.958977 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.958925    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 04:49:30.959027 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.958986    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls podName:58152a4a-74ba-4f56-a870-99a5bae3e7af nodeName:}" failed. No retries permitted until 2026-09-16 04:49:31.458967005 +0000 UTC m=+35.124896593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls") pod "dns-default-hfn7r" (UID: "58152a4a-74ba-4f56-a870-99a5bae3e7af") : secret "dns-default-metrics-tls" not found
Sep 16 04:49:30.959027 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959012    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58152a4a-74ba-4f56-a870-99a5bae3e7af-tmp-dir\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.959128 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959011    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-installation-pull-secrets\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.959128 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-trusted-ca\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.959128 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-bound-sa-token\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.959128 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959095    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-image-registry-private-configuration\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.959128 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.959338 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.959184    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 04:49:30.959338 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.959196    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675cb69c4f-hx57l: secret "image-registry-tls" not found
Sep 16 04:49:30.959338 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:30.959246    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls podName:102f3bea-c75e-4dea-9234-83c4785a68fc nodeName:}" failed. No retries permitted until 2026-09-16 04:49:31.459232387 +0000 UTC m=+35.125161976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls") pod "image-registry-675cb69c4f-hx57l" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc") : secret "image-registry-tls" not found
Sep 16 04:49:30.959338 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959271    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxv2\" (UniqueName: \"kubernetes.io/projected/5d19721b-2447-4d1f-b913-df5646e2baa0-kube-api-access-szxv2\") pod \"ingress-canary-4jtn2\" (UID: \"5d19721b-2447-4d1f-b913-df5646e2baa0\") " pod="openshift-ingress-canary/ingress-canary-4jtn2"
Sep 16 04:49:30.959338 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/102f3bea-c75e-4dea-9234-83c4785a68fc-ca-trust-extracted\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.959338 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959319    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-certificates\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.959338 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959336    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnntg\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-kube-api-access-fnntg\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.959661 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959322    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58152a4a-74ba-4f56-a870-99a5bae3e7af-config-volume\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.959661 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959380    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hpp\" (UniqueName: \"kubernetes.io/projected/58152a4a-74ba-4f56-a870-99a5bae3e7af-kube-api-access-s6hpp\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.959661 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d19721b-2447-4d1f-b913-df5646e2baa0-cert\") pod \"ingress-canary-4jtn2\" (UID: \"5d19721b-2447-4d1f-b913-df5646e2baa0\") " pod="openshift-ingress-canary/ingress-canary-4jtn2"
Sep 16 04:49:30.959661 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.959644    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/102f3bea-c75e-4dea-9234-83c4785a68fc-ca-trust-extracted\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.960089 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.960068    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-trusted-ca\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.963041 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.963020    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-installation-pull-secrets\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.963134 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.963054    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d19721b-2447-4d1f-b913-df5646e2baa0-cert\") pod \"ingress-canary-4jtn2\" (UID: \"5d19721b-2447-4d1f-b913-df5646e2baa0\") " pod="openshift-ingress-canary/ingress-canary-4jtn2"
Sep 16 04:49:30.963134 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.963077    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-image-registry-private-configuration\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.967917 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.967879    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxv2\" (UniqueName: \"kubernetes.io/projected/5d19721b-2447-4d1f-b913-df5646e2baa0-kube-api-access-szxv2\") pod \"ingress-canary-4jtn2\" (UID: \"5d19721b-2447-4d1f-b913-df5646e2baa0\") " pod="openshift-ingress-canary/ingress-canary-4jtn2"
Sep 16 04:49:30.968023 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.967947    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-bound-sa-token\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:30.968345 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.968327    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hpp\" (UniqueName: \"kubernetes.io/projected/58152a4a-74ba-4f56-a870-99a5bae3e7af-kube-api-access-s6hpp\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:30.968395 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:30.968353    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnntg\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-kube-api-access-fnntg\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:31.015684 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:31.015664    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4jtn2"
Sep 16 04:49:31.320933 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:31.320781    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4jtn2"]
Sep 16 04:49:31.324355 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:49:31.324331    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d19721b_2447_4d1f_b913_df5646e2baa0.slice/crio-632089676e72b1c06ee75884465fb5c9ef61ab2679538cc9050df0f2807c4b05 WatchSource:0}: Error finding container 632089676e72b1c06ee75884465fb5c9ef61ab2679538cc9050df0f2807c4b05: Status 404 returned error can't find the container with id 632089676e72b1c06ee75884465fb5c9ef61ab2679538cc9050df0f2807c4b05
Sep 16 04:49:31.463182 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:31.463157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:31.463278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:31.463258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:31.463323 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:31.463301    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 04:49:31.463323 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:31.463317    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675cb69c4f-hx57l: secret "image-registry-tls" not found
Sep 16 04:49:31.463393 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:31.463359    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 04:49:31.463393 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:31.463378    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls podName:102f3bea-c75e-4dea-9234-83c4785a68fc nodeName:}" failed. No retries permitted until 2026-09-16 04:49:32.46336065 +0000 UTC m=+36.129290244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls") pod "image-registry-675cb69c4f-hx57l" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc") : secret "image-registry-tls" not found
Sep 16 04:49:31.463469 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:31.463393    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls podName:58152a4a-74ba-4f56-a870-99a5bae3e7af nodeName:}" failed. No retries permitted until 2026-09-16 04:49:32.463385724 +0000 UTC m=+36.129315294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls") pod "dns-default-hfn7r" (UID: "58152a4a-74ba-4f56-a870-99a5bae3e7af") : secret "dns-default-metrics-tls" not found
Sep 16 04:49:31.780268 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:31.780240    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:49:31.782700 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:31.782668    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 04:49:31.783197 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:31.782720    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mpw9m\""
Sep 16 04:49:31.991727 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:31.991685    2573 generic.go:356] "Generic (PLEG): container finished" podID="acab9663-a074-4fbb-8596-a50e5e39b878" containerID="ce2d1ee7cd5f363d4249ff906020fba29afacc3217f8463b103c21ad0b321c49" exitCode=0
Sep 16 04:49:31.991848 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:31.991769    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tvkn" event={"ID":"acab9663-a074-4fbb-8596-a50e5e39b878","Type":"ContainerDied","Data":"ce2d1ee7cd5f363d4249ff906020fba29afacc3217f8463b103c21ad0b321c49"}
Sep 16 04:49:31.992947 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:31.992922    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4jtn2" event={"ID":"5d19721b-2447-4d1f-b913-df5646e2baa0","Type":"ContainerStarted","Data":"632089676e72b1c06ee75884465fb5c9ef61ab2679538cc9050df0f2807c4b05"}
Sep 16 04:49:32.470978 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:32.470900    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:32.470978 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:32.470962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:32.471175 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:32.471066    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 04:49:32.471175 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:32.471100    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 04:49:32.471175 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:32.471115    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675cb69c4f-hx57l: secret "image-registry-tls" not found
Sep 16 04:49:32.471175 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:32.471139    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls podName:58152a4a-74ba-4f56-a870-99a5bae3e7af nodeName:}" failed. No retries permitted until 2026-09-16 04:49:34.471120388 +0000 UTC m=+38.137049990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls") pod "dns-default-hfn7r" (UID: "58152a4a-74ba-4f56-a870-99a5bae3e7af") : secret "dns-default-metrics-tls" not found
Sep 16 04:49:32.471175 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:32.471157    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls podName:102f3bea-c75e-4dea-9234-83c4785a68fc nodeName:}" failed. No retries permitted until 2026-09-16 04:49:34.471145927 +0000 UTC m=+38.137075516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls") pod "image-registry-675cb69c4f-hx57l" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc") : secret "image-registry-tls" not found
Sep 16 04:49:32.999409 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:32.999376    2573 generic.go:356] "Generic (PLEG): container finished" podID="acab9663-a074-4fbb-8596-a50e5e39b878" containerID="dc322427cf8800c94d0c2169eee6907ee1bda325ea8d8c8e4c5dbca84cbd02b8" exitCode=0
Sep 16 04:49:32.999802 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:32.999424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tvkn" event={"ID":"acab9663-a074-4fbb-8596-a50e5e39b878","Type":"ContainerDied","Data":"dc322427cf8800c94d0c2169eee6907ee1bda325ea8d8c8e4c5dbca84cbd02b8"}
Sep 16 04:49:33.135298 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:33.135263    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:33.630937 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:33.630891    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:33.726109 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:33.726087    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:34.005684 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.005653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tvkn" event={"ID":"acab9663-a074-4fbb-8596-a50e5e39b878","Type":"ContainerStarted","Data":"856d2b0587ce474ab83d16830041729032399278bfd210c9e2598b6821d83881"}
Sep 16 04:49:34.007239 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.007215    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4jtn2" event={"ID":"5d19721b-2447-4d1f-b913-df5646e2baa0","Type":"ContainerStarted","Data":"eae0615a07c404ae419f96a63e6a6ea70a4c1c29ec782ca83d5eb1d23bca43c1"}
Sep 16 04:49:34.028739 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.028693    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2tvkn" podStartSLOduration=9.074659858 podStartE2EDuration="37.028679497s" totalImagesPullingTime="27.954019639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:34.0277943 +0000 UTC m=+37.693723913" watchObservedRunningTime="2026-09-16 04:49:34.028679497 +0000 UTC m=+37.694609088"
Sep 16 04:49:34.044038 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.043997    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4jtn2" podStartSLOduration=1.534620941 podStartE2EDuration="4.043984133s" totalImagesPullingTime="2.509363192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:34.043361072 +0000 UTC m=+37.709290697" watchObservedRunningTime="2026-09-16 04:49:34.043984133 +0000 UTC m=+37.709913763"
Sep 16 04:49:34.120538 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.120341    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:34.178922 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.178898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:34.403237 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.403185    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:34.459666 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.459643    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:34.484626 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.484588    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:34.484706 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.484686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:34.484761 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:34.484711    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 04:49:34.484761 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:34.484724    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675cb69c4f-hx57l: secret "image-registry-tls" not found
Sep 16 04:49:34.484859 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:34.484772    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls podName:102f3bea-c75e-4dea-9234-83c4785a68fc nodeName:}" failed. No retries permitted until 2026-09-16 04:49:38.484758206 +0000 UTC m=+42.150687776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls") pod "image-registry-675cb69c4f-hx57l" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc") : secret "image-registry-tls" not found
Sep 16 04:49:34.484859 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:34.484778    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 04:49:34.484859 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:34.484826    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls podName:58152a4a-74ba-4f56-a870-99a5bae3e7af nodeName:}" failed. No retries permitted until 2026-09-16 04:49:38.484811119 +0000 UTC m=+42.150740705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls") pod "dns-default-hfn7r" (UID: "58152a4a-74ba-4f56-a870-99a5bae3e7af") : secret "dns-default-metrics-tls" not found
Sep 16 04:49:34.579103 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.579081    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:34.630943 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:34.630920    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:36.053885 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:36.053857    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:38.513571 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:38.513537    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:38.513998 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:38.513612    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:38.513998 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:38.513696    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 04:49:38.513998 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:38.513722    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675cb69c4f-hx57l: secret "image-registry-tls" not found
Sep 16 04:49:38.513998 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:38.513699    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 04:49:38.513998 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:38.513784    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls podName:102f3bea-c75e-4dea-9234-83c4785a68fc nodeName:}" failed. No retries permitted until 2026-09-16 04:49:46.513763833 +0000 UTC m=+50.179693447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls") pod "image-registry-675cb69c4f-hx57l" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc") : secret "image-registry-tls" not found
Sep 16 04:49:38.513998 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:38.513803    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls podName:58152a4a-74ba-4f56-a870-99a5bae3e7af nodeName:}" failed. No retries permitted until 2026-09-16 04:49:46.513794254 +0000 UTC m=+50.179723872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls") pod "dns-default-hfn7r" (UID: "58152a4a-74ba-4f56-a870-99a5bae3e7af") : secret "dns-default-metrics-tls" not found
Sep 16 04:49:40.286485 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.286451    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q7ss4"]
Sep 16 04:49:40.291261 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.291231    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:40.319056 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.319033    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q7ss4"]
Sep 16 04:49:40.319141 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.319127    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.321473 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.321444    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 04:49:40.321585 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.321485    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 04:49:40.321585 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.321496    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jkwpt\""
Sep 16 04:49:40.321585 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.321509    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 04:49:40.321813 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.321795    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 04:49:40.360563 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.360539    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:40.383743 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.383723    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6"]
Sep 16 04:49:40.404956 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.404939    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6"
Sep 16 04:49:40.404956 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.404949    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6"]
Sep 16 04:49:40.408909 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.408893    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 04:49:40.409162 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.409143    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-p74sx\""
Sep 16 04:49:40.409263 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.409185    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 04:49:40.425934 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.425917    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lp6w\" (UniqueName: \"kubernetes.io/projected/b7b38734-74c5-457e-9b84-32938c288a83-kube-api-access-2lp6w\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.426020 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.425941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7b38734-74c5-457e-9b84-32938c288a83-crio-socket\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.426020 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.425978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.426093 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.426061    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7b38734-74c5-457e-9b84-32938c288a83-data-volume\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.426093 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.426087    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.450542 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.450518    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:40.526720 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.526699    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lp6w\" (UniqueName: \"kubernetes.io/projected/b7b38734-74c5-457e-9b84-32938c288a83-kube-api-access-2lp6w\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.526800 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.526727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7b38734-74c5-457e-9b84-32938c288a83-crio-socket\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.526800 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.526774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.526867 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.526826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bkl\" (UniqueName: \"kubernetes.io/projected/50c070c3-26e5-4761-8655-cf31cbc767df-kube-api-access-78bkl\") pod \"migrator-8b4d697c5-j8nd6\" (UID: \"50c070c3-26e5-4761-8655-cf31cbc767df\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6"
Sep 16 04:49:40.526867 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.526859    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7b38734-74c5-457e-9b84-32938c288a83-data-volume\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.526940 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.526886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.526940 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.526922    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7b38734-74c5-457e-9b84-32938c288a83-crio-socket\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.526940 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:40.526934    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 04:49:40.527046 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:40.526946    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 04:49:40.527046 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:40.527012    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm podName:b7b38734-74c5-457e-9b84-32938c288a83 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:41.026999345 +0000 UTC m=+44.692928921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q7ss4" (UID: "b7b38734-74c5-457e-9b84-32938c288a83") : configmap "kube-rbac-proxy" not found
Sep 16 04:49:40.527046 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:40.527026    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls podName:b7b38734-74c5-457e-9b84-32938c288a83 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:41.027019531 +0000 UTC m=+44.692949100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q7ss4" (UID: "b7b38734-74c5-457e-9b84-32938c288a83") : secret "insights-runtime-extractor-tls" not found
Sep 16 04:49:40.527193 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.527165    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7b38734-74c5-457e-9b84-32938c288a83-data-volume\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.537834 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.537782    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lp6w\" (UniqueName: \"kubernetes.io/projected/b7b38734-74c5-457e-9b84-32938c288a83-kube-api-access-2lp6w\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:40.615879 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.615858    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:40.628072 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.628052    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bkl\" (UniqueName: \"kubernetes.io/projected/50c070c3-26e5-4761-8655-cf31cbc767df-kube-api-access-78bkl\") pod \"migrator-8b4d697c5-j8nd6\" (UID: \"50c070c3-26e5-4761-8655-cf31cbc767df\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6"
Sep 16 04:49:40.635088 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.635065    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bkl\" (UniqueName: \"kubernetes.io/projected/50c070c3-26e5-4761-8655-cf31cbc767df-kube-api-access-78bkl\") pod \"migrator-8b4d697c5-j8nd6\" (UID: \"50c070c3-26e5-4761-8655-cf31cbc767df\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6"
Sep 16 04:49:40.672362 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.672338    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:40.712546 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.712525    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6"
Sep 16 04:49:40.988329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:40.988303    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6"]
Sep 16 04:49:40.991494 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:49:40.991466    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c070c3_26e5_4761_8655_cf31cbc767df.slice/crio-b0dc44804dd5f6591a96b26cde081f60c48b54f5b524365ed13006e60465442a WatchSource:0}: Error finding container b0dc44804dd5f6591a96b26cde081f60c48b54f5b524365ed13006e60465442a: Status 404 returned error can't find the container with id b0dc44804dd5f6591a96b26cde081f60c48b54f5b524365ed13006e60465442a
Sep 16 04:49:41.023350 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:41.023323    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6" event={"ID":"50c070c3-26e5-4761-8655-cf31cbc767df","Type":"ContainerStarted","Data":"b0dc44804dd5f6591a96b26cde081f60c48b54f5b524365ed13006e60465442a"}
Sep 16 04:49:41.033635 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:41.033610    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:41.033714 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:41.033667    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:41.033714 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:41.033685    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 04:49:41.033779 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:41.033737    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls podName:b7b38734-74c5-457e-9b84-32938c288a83 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:42.033723395 +0000 UTC m=+45.699652989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q7ss4" (UID: "b7b38734-74c5-457e-9b84-32938c288a83") : secret "insights-runtime-extractor-tls" not found
Sep 16 04:49:41.033779 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:41.033741    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 04:49:41.033852 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:41.033783    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm podName:b7b38734-74c5-457e-9b84-32938c288a83 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:42.03377282 +0000 UTC m=+45.699702390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q7ss4" (UID: "b7b38734-74c5-457e-9b84-32938c288a83") : configmap "kube-rbac-proxy" not found
Sep 16 04:49:41.296124 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:41.296071    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:41.385485 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:41.385465    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:41.638621 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:41.638586    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-69szj_64223cf5-19e6-4f4a-aa87-67766e0373d9/dns-node-resolver/0.log"
Sep 16 04:49:42.040241 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:42.040171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:42.040378 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:42.040251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:42.040378 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:42.040285    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 04:49:42.040378 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:42.040340    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm podName:b7b38734-74c5-457e-9b84-32938c288a83 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.040326674 +0000 UTC m=+47.706256272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q7ss4" (UID: "b7b38734-74c5-457e-9b84-32938c288a83") : configmap "kube-rbac-proxy" not found
Sep 16 04:49:42.040554 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:42.040386    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 04:49:42.040554 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:42.040434    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls podName:b7b38734-74c5-457e-9b84-32938c288a83 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.040423145 +0000 UTC m=+47.706352735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q7ss4" (UID: "b7b38734-74c5-457e-9b84-32938c288a83") : secret "insights-runtime-extractor-tls" not found
Sep 16 04:49:42.202963 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:42.202933    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:42.259232 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:42.259208    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:42.839040 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:42.839024    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rw649_01b6aa0b-369b-4585-bcbc-890512014439/node-ca/0.log"
Sep 16 04:49:42.957372 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:42.957317    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:43.028687 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:43.028658    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6" event={"ID":"50c070c3-26e5-4761-8655-cf31cbc767df","Type":"ContainerStarted","Data":"1452427a56315408e5f1750d87fcc0c67bbf743fc4b6175181a257baeb7a652c"}
Sep 16 04:49:43.028778 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:43.028706    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6" event={"ID":"50c070c3-26e5-4761-8655-cf31cbc767df","Type":"ContainerStarted","Data":"2bdbd13f5054513db87fec5a6b10c18211407fcd080acd90b343ffd151b98142"}
Sep 16 04:49:43.044112 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:43.044069    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-j8nd6" podStartSLOduration=1.333594337 podStartE2EDuration="3.044055655s" totalImagesPullingTime="1.710461318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:43.042863163 +0000 UTC m=+46.708792770" watchObservedRunningTime="2026-09-16 04:49:43.044055655 +0000 UTC m=+46.709985247"
Sep 16 04:49:43.124592 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:43.124552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:43.182158 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:43.182132    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:43.440153 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:43.440130    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4jtn2_5d19721b-2447-4d1f-b913-df5646e2baa0/serve-healthcheck-canary/0.log"
Sep 16 04:49:43.851771 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:43.851715    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:44.056489 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:44.056468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:44.056613 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:44.056536    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:44.056660 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:44.056605    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 04:49:44.056693 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:44.056668    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm podName:b7b38734-74c5-457e-9b84-32938c288a83 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:48.056653963 +0000 UTC m=+51.722583563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q7ss4" (UID: "b7b38734-74c5-457e-9b84-32938c288a83") : configmap "kube-rbac-proxy" not found
Sep 16 04:49:44.056735 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:44.056688    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 04:49:44.056773 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:44.056748    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls podName:b7b38734-74c5-457e-9b84-32938c288a83 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:48.056729026 +0000 UTC m=+51.722658641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q7ss4" (UID: "b7b38734-74c5-457e-9b84-32938c288a83") : secret "insights-runtime-extractor-tls" not found
Sep 16 04:49:45.036181 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:45.036143    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:46.024172 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:46.024140    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:46.576916 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:46.576883    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:49:46.577333 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:46.576965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:49:46.577333 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:46.577062    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 04:49:46.577333 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:46.577077    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 04:49:46.577333 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:46.577101    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675cb69c4f-hx57l: secret "image-registry-tls" not found
Sep 16 04:49:46.577333 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:46.577114    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls podName:58152a4a-74ba-4f56-a870-99a5bae3e7af nodeName:}" failed. No retries permitted until 2026-09-16 04:50:02.577100745 +0000 UTC m=+66.243030397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls") pod "dns-default-hfn7r" (UID: "58152a4a-74ba-4f56-a870-99a5bae3e7af") : secret "dns-default-metrics-tls" not found
Sep 16 04:49:46.577333 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:46.577161    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls podName:102f3bea-c75e-4dea-9234-83c4785a68fc nodeName:}" failed. No retries permitted until 2026-09-16 04:50:02.57714142 +0000 UTC m=+66.243071042 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls") pod "image-registry-675cb69c4f-hx57l" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc") : secret "image-registry-tls" not found
Sep 16 04:49:46.681906 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:46.681878    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:47.088984 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:47.088953    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:47.279240 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:47.279212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:47.411691 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:47.411666    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:47.511580 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:47.511557    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:47.611904 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:47.611876    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:47.713048 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:47.712995    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:47.818206 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:47.818180    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:48.087180 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:48.087117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:48.087285 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:48.087213    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:48.087368 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:48.087352    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 04:49:48.087430 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:49:48.087420    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls podName:b7b38734-74c5-457e-9b84-32938c288a83 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:56.087402668 +0000 UTC m=+59.753332257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q7ss4" (UID: "b7b38734-74c5-457e-9b84-32938c288a83") : secret "insights-runtime-extractor-tls" not found
Sep 16 04:49:48.087707 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:48.087685    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b38734-74c5-457e-9b84-32938c288a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:48.777765 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:48.777737    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:49.178819 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:49.178786    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:49.383766 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:49.383724    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:49.583312 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:49.583285    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:49.980333 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:49.980283    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:50.232723 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:50.232668    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:50.578320 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:50.578263    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:50.782080 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:50.782044    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:51.025895 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:51.025871    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:51.127458 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:51.127425    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:51.380185 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:51.380161    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:51.780491 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:51.780427    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:51.834694 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:51.834664    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:51.979082 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:51.979064    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:52.383176 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:52.383151    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:52.484174 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:52.484149    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:52.535771 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:52.535747    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:52.981409 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:52.981384    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:53.178201 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:53.178176    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:53.288087 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:53.288015    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:53.590050 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:53.590004    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:54.379394 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:54.379370    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:54.776286 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:54.776219    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:54.982001 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:54.981976    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:55.183083 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:55.183063    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:55.579650 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:55.579556    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:55.984145 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:55.984114    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4q88"
Sep 16 04:49:56.145122 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:56.145091    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:56.147446 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:56.147421    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b38734-74c5-457e-9b84-32938c288a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7ss4\" (UID: \"b7b38734-74c5-457e-9b84-32938c288a83\") " pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:56.180394 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:56.180366    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:56.228762 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:56.228732    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q7ss4"
Sep 16 04:49:56.502671 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:56.502586    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q7ss4"]
Sep 16 04:49:56.785099 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:56.785034    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:56.980362 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:56.980329    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:57.064810 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:57.064744    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q7ss4" event={"ID":"b7b38734-74c5-457e-9b84-32938c288a83","Type":"ContainerStarted","Data":"5fc85ac7ed00f0d74ce3b74ec765ef10bbedf9a3754ad3b938db6325779752d7"}
Sep 16 04:49:57.064810 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:57.064801    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q7ss4" event={"ID":"b7b38734-74c5-457e-9b84-32938c288a83","Type":"ContainerStarted","Data":"db96fd4ea5a5b269fc3291f865caf92842575145994424163c279bab8038db88"}
Sep 16 04:49:57.204922 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:57.204900    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:57.304526 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:57.304503    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:57.388120 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:57.388100    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:57.582326 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:57.582301    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:57.807218 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:57.807155    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:49:58.069742 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:58.069679    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q7ss4" event={"ID":"b7b38734-74c5-457e-9b84-32938c288a83","Type":"ContainerStarted","Data":"4a2425890a0f712cb8f27d04b247807f0d436d67b1e6a6767dc7e2b41221eb90"}
Sep 16 04:49:59.075170 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:59.075132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q7ss4" event={"ID":"b7b38734-74c5-457e-9b84-32938c288a83","Type":"ContainerStarted","Data":"1ee900c60cdd46601b8aba8549b16096889f1bba58070bc3beacbb87a0e791fd"}
Sep 16 04:49:59.092505 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:49:59.092335    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q7ss4" podStartSLOduration=17.136146722 podStartE2EDuration="19.092316519s" totalImagesPullingTime="1.956169797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:49:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:59.091700491 +0000 UTC m=+62.757630093" watchObservedRunningTime="2026-09-16 04:49:59.092316519 +0000 UTC m=+62.758246112"
Sep 16 04:50:02.593588 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.593549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:50:02.594077 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.593630    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:50:02.594077 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.593741    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:50:02.594077 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.593805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:50:02.595917 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.595894    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 04:50:02.596067 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.595936    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 04:50:02.596284 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.596253    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") pod \"image-registry-675cb69c4f-hx57l\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") " pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:50:02.596336 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.596291    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58152a4a-74ba-4f56-a870-99a5bae3e7af-metrics-tls\") pod \"dns-default-hfn7r\" (UID: \"58152a4a-74ba-4f56-a870-99a5bae3e7af\") " pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:50:02.606532 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.606500    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9705e01-c8d0-481f-8769-ffd767bc202c-original-pull-secret\") pod \"global-pull-secret-syncer-vqmw5\" (UID: \"b9705e01-c8d0-481f-8769-ffd767bc202c\") " pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:50:02.606660 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.606500    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f4ee4-d6b0-4fb6-b6e6-86277d856c67-metrics-certs\") pod \"network-metrics-daemon-k449w\" (UID: \"835f4ee4-d6b0-4fb6-b6e6-86277d856c67\") " pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:50:02.694102 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.694078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv87\" (UniqueName: \"kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87\") pod \"network-check-target-64c4f\" (UID: \"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5\") " pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:50:02.696120 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.696095    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mpw9m\""
Sep 16 04:50:02.696219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.696136    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 04:50:02.704706 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.704687    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k449w"
Sep 16 04:50:02.706212 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.706193    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 04:50:02.717999 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.717976    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsv87\" (UniqueName: \"kubernetes.io/projected/c0c4f544-8e7a-4ffe-bd00-96cc6057edc5-kube-api-access-hsv87\") pod \"network-check-target-64c4f\" (UID: \"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5\") " pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:50:02.792119 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.792096    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tff4l\""
Sep 16 04:50:02.800716 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.800700    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:50:02.837935 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.837913    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgbjt\""
Sep 16 04:50:02.846794 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.846769    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:50:02.894345 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.894118    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gbd6v\""
Sep 16 04:50:02.896850 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.896827    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqmw5"
Sep 16 04:50:02.901433 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.901409    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:50:02.988721 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:02.988690    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k449w"]
Sep 16 04:50:02.992233 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:50:02.992212    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835f4ee4_d6b0_4fb6_b6e6_86277d856c67.slice/crio-33cf12ea2221928361e149f6c4fe40e2af9ed857be0242ce11a6ea0c505bd234 WatchSource:0}: Error finding container 33cf12ea2221928361e149f6c4fe40e2af9ed857be0242ce11a6ea0c505bd234: Status 404 returned error can't find the container with id 33cf12ea2221928361e149f6c4fe40e2af9ed857be0242ce11a6ea0c505bd234
Sep 16 04:50:03.067421 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:03.067399    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675cb69c4f-hx57l"]
Sep 16 04:50:03.070157 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:50:03.070138    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102f3bea_c75e_4dea_9234_83c4785a68fc.slice/crio-65bd43ed4e8dcc30678ae1010ef63c24c6ed8b5bbcbe25bea6a909dc6624f1ec WatchSource:0}: Error finding container 65bd43ed4e8dcc30678ae1010ef63c24c6ed8b5bbcbe25bea6a909dc6624f1ec: Status 404 returned error can't find the container with id 65bd43ed4e8dcc30678ae1010ef63c24c6ed8b5bbcbe25bea6a909dc6624f1ec
Sep 16 04:50:03.088328 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:03.088302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l" event={"ID":"102f3bea-c75e-4dea-9234-83c4785a68fc","Type":"ContainerStarted","Data":"65bd43ed4e8dcc30678ae1010ef63c24c6ed8b5bbcbe25bea6a909dc6624f1ec"}
Sep 16 04:50:03.089339 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:03.089317    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k449w" event={"ID":"835f4ee4-d6b0-4fb6-b6e6-86277d856c67","Type":"ContainerStarted","Data":"33cf12ea2221928361e149f6c4fe40e2af9ed857be0242ce11a6ea0c505bd234"}
Sep 16 04:50:03.114729 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:03.114711    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hfn7r"]
Sep 16 04:50:03.117376 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:50:03.117353    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58152a4a_74ba_4f56_a870_99a5bae3e7af.slice/crio-2e3653b93ae1b610e09787a986d4e713a13e293c060c8c9a4acee03baeddbf16 WatchSource:0}: Error finding container 2e3653b93ae1b610e09787a986d4e713a13e293c060c8c9a4acee03baeddbf16: Status 404 returned error can't find the container with id 2e3653b93ae1b610e09787a986d4e713a13e293c060c8c9a4acee03baeddbf16
Sep 16 04:50:03.160769 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:03.160748    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vqmw5"]
Sep 16 04:50:03.163760 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:50:03.163734    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9705e01_c8d0_481f_8769_ffd767bc202c.slice/crio-3d40eccfdcbd70e7d868f946f90e6d405f570f6acd62e2a536c13c9f4525e1fe WatchSource:0}: Error finding container 3d40eccfdcbd70e7d868f946f90e6d405f570f6acd62e2a536c13c9f4525e1fe: Status 404 returned error can't find the container with id 3d40eccfdcbd70e7d868f946f90e6d405f570f6acd62e2a536c13c9f4525e1fe
Sep 16 04:50:03.163921 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:03.163897    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-64c4f"]
Sep 16 04:50:04.098368 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:04.098237    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l" event={"ID":"102f3bea-c75e-4dea-9234-83c4785a68fc","Type":"ContainerStarted","Data":"543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9"}
Sep 16 04:50:04.098368 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:04.098314    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:50:04.100174 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:04.100145    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-64c4f" event={"ID":"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5","Type":"ContainerStarted","Data":"684becda1f6055fba766dd6f2af97edb00ff53af9ce99a6329e94c624f9ae62c"}
Sep 16 04:50:04.101563 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:04.101524    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vqmw5" event={"ID":"b9705e01-c8d0-481f-8769-ffd767bc202c","Type":"ContainerStarted","Data":"3d40eccfdcbd70e7d868f946f90e6d405f570f6acd62e2a536c13c9f4525e1fe"}
Sep 16 04:50:04.104379 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:04.104354    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hfn7r" event={"ID":"58152a4a-74ba-4f56-a870-99a5bae3e7af","Type":"ContainerStarted","Data":"2e3653b93ae1b610e09787a986d4e713a13e293c060c8c9a4acee03baeddbf16"}
Sep 16 04:50:04.117546 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:04.117509    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l" podStartSLOduration=62.117491275 podStartE2EDuration="1m2.117491275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:04.116422533 +0000 UTC m=+67.782352150" watchObservedRunningTime="2026-09-16 04:50:04.117491275 +0000 UTC m=+67.783420872"
Sep 16 04:50:05.113387 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.113342    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k449w" event={"ID":"835f4ee4-d6b0-4fb6-b6e6-86277d856c67","Type":"ContainerStarted","Data":"b03badbd59648aaa58d8f2429d3728c719f2b03fc81302820d4fc95f47568342"}
Sep 16 04:50:05.113748 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.113439    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k449w" event={"ID":"835f4ee4-d6b0-4fb6-b6e6-86277d856c67","Type":"ContainerStarted","Data":"c80c591444a6ab8996b9beb509c798a272f0f5bc72a227162c1185fbaf011261"}
Sep 16 04:50:05.129310 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.129269    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k449w" podStartSLOduration=67.017677831 podStartE2EDuration="1m8.129254431s" totalImagesPullingTime="1.1115766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:05.127980427 +0000 UTC m=+68.793910048" watchObservedRunningTime="2026-09-16 04:50:05.129254431 +0000 UTC m=+68.795184079"
Sep 16 04:50:05.556367 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.556334    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675cb69c4f-hx57l"]
Sep 16 04:50:05.572315 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.571591    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"]
Sep 16 04:50:05.574273 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.574245    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"]
Sep 16 04:50:05.574407 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.574387    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.576360 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.576300    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.576628 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.576577    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 04:50:05.576916 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.576895    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 04:50:05.577030 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.576942    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 04:50:05.577264 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.577248    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 04:50:05.581095 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.580664    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 04:50:05.581095 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.580716    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 04:50:05.581095 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.580725    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 04:50:05.581095 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.580780    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 04:50:05.585311 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.585266    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"]
Sep 16 04:50:05.592012 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.591965    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"]
Sep 16 04:50:05.615294 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.615268    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.615400 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.615303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.615400 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.615369    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/55796606-d07b-4c02-be31-c1d17d66b250-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b57b7b46c-2nf5q\" (UID: \"55796606-d07b-4c02-be31-c1d17d66b250\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.615509 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.615409    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4x7\" (UniqueName: \"kubernetes.io/projected/55796606-d07b-4c02-be31-c1d17d66b250-kube-api-access-9b4x7\") pod \"klusterlet-addon-workmgr-5b57b7b46c-2nf5q\" (UID: \"55796606-d07b-4c02-be31-c1d17d66b250\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.615509 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.615469    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-ca\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.615509 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.615500    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55796606-d07b-4c02-be31-c1d17d66b250-tmp\") pod \"klusterlet-addon-workmgr-5b57b7b46c-2nf5q\" (UID: \"55796606-d07b-4c02-be31-c1d17d66b250\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.615724 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.615529    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.615724 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.615556    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvgr\" (UniqueName: \"kubernetes.io/projected/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-kube-api-access-9rvgr\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.615724 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.615579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-hub\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.711243 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.711213    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:05.716149 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.716126    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/55796606-d07b-4c02-be31-c1d17d66b250-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b57b7b46c-2nf5q\" (UID: \"55796606-d07b-4c02-be31-c1d17d66b250\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.716261 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.716166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4x7\" (UniqueName: \"kubernetes.io/projected/55796606-d07b-4c02-be31-c1d17d66b250-kube-api-access-9b4x7\") pod \"klusterlet-addon-workmgr-5b57b7b46c-2nf5q\" (UID: \"55796606-d07b-4c02-be31-c1d17d66b250\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.716261 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.716207    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-ca\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.716382 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.716258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55796606-d07b-4c02-be31-c1d17d66b250-tmp\") pod \"klusterlet-addon-workmgr-5b57b7b46c-2nf5q\" (UID: \"55796606-d07b-4c02-be31-c1d17d66b250\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.716382 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.716301    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.716382 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.716329    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvgr\" (UniqueName: \"kubernetes.io/projected/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-kube-api-access-9rvgr\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.716382 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.716356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-hub\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.716572 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.716389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.716572 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.716429    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.717006 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.716984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55796606-d07b-4c02-be31-c1d17d66b250-tmp\") pod \"klusterlet-addon-workmgr-5b57b7b46c-2nf5q\" (UID: \"55796606-d07b-4c02-be31-c1d17d66b250\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.717144 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.717124    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.719167 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.719143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/55796606-d07b-4c02-be31-c1d17d66b250-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b57b7b46c-2nf5q\" (UID: \"55796606-d07b-4c02-be31-c1d17d66b250\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.719730 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.719684    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.720219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.720175    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-hub\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.720324 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.720265    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.720484 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.720466    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-ca\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.728267 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.728211    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4x7\" (UniqueName: \"kubernetes.io/projected/55796606-d07b-4c02-be31-c1d17d66b250-kube-api-access-9b4x7\") pod \"klusterlet-addon-workmgr-5b57b7b46c-2nf5q\" (UID: \"55796606-d07b-4c02-be31-c1d17d66b250\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.731118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.731079    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvgr\" (UniqueName: \"kubernetes.io/projected/fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88-kube-api-access-9rvgr\") pod \"cluster-proxy-proxy-agent-5d749c67f6-7r88q\" (UID: \"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.837234 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.837169    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:05.887664 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.887637    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:05.904367 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.904348    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"
Sep 16 04:50:05.932886 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:05.932862    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:06.120007 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:06.119926    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hfn7r" event={"ID":"58152a4a-74ba-4f56-a870-99a5bae3e7af","Type":"ContainerStarted","Data":"d24282b71a284ca5ae5ec8bd0b78374dbbb55c376eac4426a4c698a67f8e490c"}
Sep 16 04:50:06.120007 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:06.119991    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hfn7r" event={"ID":"58152a4a-74ba-4f56-a870-99a5bae3e7af","Type":"ContainerStarted","Data":"5b9bb50f076bcc560311f9f7c02af0f1de68be7572153400df7cef745d2d65d2"}
Sep 16 04:50:06.120458 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:06.120335    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:50:06.142730 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:06.142676    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hfn7r" podStartSLOduration=34.228826816 podStartE2EDuration="36.142659433s" totalImagesPullingTime="1.913832617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:06.141790477 +0000 UTC m=+69.807720098" watchObservedRunningTime="2026-09-16 04:50:06.142659433 +0000 UTC m=+69.808589021"
Sep 16 04:50:06.615356 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:06.615318    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:06.672229 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:06.672199    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:06.990629 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:06.990587    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:07.094652 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:07.094626    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:07.191870 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:07.191831    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:07.309507 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:07.309426    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:07.564803 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:07.564724    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:07.634691 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:07.634657    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:07.705894 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:07.705866    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:07.859060 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:07.859001    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:08.000496 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.000468    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:08.101338 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.101306    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:08.128582 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.128552    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-64c4f" event={"ID":"c0c4f544-8e7a-4ffe-bd00-96cc6057edc5","Type":"ContainerStarted","Data":"d185eb3b02f9c3affa96a44162e54b3ca46f6f57da2d9fec32610d7211a72827"}
Sep 16 04:50:08.128741 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.128667    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:50:08.130170 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.130147    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vqmw5" event={"ID":"b9705e01-c8d0-481f-8769-ffd767bc202c","Type":"ContainerStarted","Data":"0cd26fcf8a6df123314fdd6051be6398846cd1bc82775c4b655f96bda1716040"}
Sep 16 04:50:08.145381 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.145340    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-64c4f" podStartSLOduration=66.396389197 podStartE2EDuration="1m11.145328399s" totalImagesPullingTime="4.748939202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:08.14382236 +0000 UTC m=+71.809751962" watchObservedRunningTime="2026-09-16 04:50:08.145328399 +0000 UTC m=+71.811258299"
Sep 16 04:50:08.157469 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.157424    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vqmw5" podStartSLOduration=66.390627212 podStartE2EDuration="1m11.157410454s" totalImagesPullingTime="4.766783242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:08.157002569 +0000 UTC m=+71.822932171" watchObservedRunningTime="2026-09-16 04:50:08.157410454 +0000 UTC m=+71.823340047"
Sep 16 04:50:08.190237 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.190208    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"]
Sep 16 04:50:08.192671 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.192650    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q"]
Sep 16 04:50:08.194314 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:50:08.194291    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55796606_d07b_4c02_be31_c1d17d66b250.slice/crio-b97f2407860c0108e27e31e8130c1118765650abddf780e37c18b07033dd72b6 WatchSource:0}: Error finding container b97f2407860c0108e27e31e8130c1118765650abddf780e37c18b07033dd72b6: Status 404 returned error can't find the container with id b97f2407860c0108e27e31e8130c1118765650abddf780e37c18b07033dd72b6
Sep 16 04:50:08.195702 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:50:08.195680    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb3df2ae_ed54_4c9c_8b20_d8f4a2e02c88.slice/crio-a4d823ed4f49292ed4443070d9eea2567775aff98822d8f80637e637fdb39e79 WatchSource:0}: Error finding container a4d823ed4f49292ed4443070d9eea2567775aff98822d8f80637e637fdb39e79: Status 404 returned error can't find the container with id a4d823ed4f49292ed4443070d9eea2567775aff98822d8f80637e637fdb39e79
Sep 16 04:50:08.240190 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.240173    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:08.400237 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.400183    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:08.497179 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.497156    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:08.849321 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.849244    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:08.909114 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:08.909080    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:09.043539 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:09.043509    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:09.136105 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:09.136073    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q" event={"ID":"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88","Type":"ContainerStarted","Data":"a4d823ed4f49292ed4443070d9eea2567775aff98822d8f80637e637fdb39e79"}
Sep 16 04:50:09.138152 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:09.138127    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q" event={"ID":"55796606-d07b-4c02-be31-c1d17d66b250","Type":"ContainerStarted","Data":"b97f2407860c0108e27e31e8130c1118765650abddf780e37c18b07033dd72b6"}
Sep 16 04:50:09.277667 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:09.277617    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:09.443897 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:09.443725    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:11.064448 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:11.064403    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:11.330648 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:11.330558    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:11.400534 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:11.400493    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:11.676237 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:11.676207    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:12.204559 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:12.204525    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:12.260139 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:12.260112    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:14.157990 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.157959    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q" event={"ID":"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88","Type":"ContainerStarted","Data":"0514596bee6dc0f2b38bdfa35f4b77a8ac8d85ba4407f22abea2df8e11321266"}
Sep 16 04:50:14.158380 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.158011    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q" event={"ID":"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88","Type":"ContainerStarted","Data":"709d3df146a1e1f91661272bf3a459febc6400014c2156cf58c7d334f6bbe779"}
Sep 16 04:50:14.158380 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.158032    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q" event={"ID":"fb3df2ae-ed54-4c9c-8b20-d8f4a2e02c88","Type":"ContainerStarted","Data":"4f2f95cf8103531805f7bf9c5206e8771ae9cc897e1e296bdf2227ec0c9a52d5"}
Sep 16 04:50:14.159389 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.159366    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q" event={"ID":"55796606-d07b-4c02-be31-c1d17d66b250","Type":"ContainerStarted","Data":"aec3e8c803c5ed15ef991df3b5a3fd4b133ed0f1365c8fa1dc06bd7ebf2d2812"}
Sep 16 04:50:14.159568 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.159553    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:14.161258 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.161239    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q"
Sep 16 04:50:14.177555 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.177516    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d749c67f6-7r88q" podStartSLOduration=3.743031619 podStartE2EDuration="9.177504085s" totalImagesPullingTime="5.434472466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:14.176138205 +0000 UTC m=+77.842067807" watchObservedRunningTime="2026-09-16 04:50:14.177504085 +0000 UTC m=+77.843433677"
Sep 16 04:50:14.190820 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.190783    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b57b7b46c-2nf5q" podStartSLOduration=3.75271152 podStartE2EDuration="9.190768738s" totalImagesPullingTime="5.438057218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:14.189885866 +0000 UTC m=+77.855815458" watchObservedRunningTime="2026-09-16 04:50:14.190768738 +0000 UTC m=+77.856698329"
Sep 16 04:50:14.368132 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.368106    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:14.468339 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.468274    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:14.523952 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:14.523925    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:17.133412 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:17.133378    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hfn7r"
Sep 16 04:50:18.500097 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:18.500057    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:19.297502 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:19.297472    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:19.398521 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:19.398494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:21.488024 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.487992    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:21.761658 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.761562    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:21.796687 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.796653    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dqswl"]
Sep 16 04:50:21.800035 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.800014    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:21.802829 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.802802    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 04:50:21.802933 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.802829    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 04:50:21.803407 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.803381    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 04:50:21.803545 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.803416    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k8987\""
Sep 16 04:50:21.803545 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.803456    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 04:50:21.803545 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.803520    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 04:50:21.803712 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.803523    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 04:50:21.859588 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.859564    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:21.929819 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.929794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvzs2\" (UniqueName: \"kubernetes.io/projected/15c4e760-06e6-4df7-8cea-2cc464744f78-kube-api-access-lvzs2\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:21.929920 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.929827    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-accelerators-collector-config\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:21.929920 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.929851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-wtmp\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:21.929920 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.929898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:21.930012 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.929926    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15c4e760-06e6-4df7-8cea-2cc464744f78-sys\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:21.930012 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.929946    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15c4e760-06e6-4df7-8cea-2cc464744f78-root\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:21.930012 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.929995    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-tls\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:21.930101 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.930024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-textfile\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:21.930101 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.930048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c4e760-06e6-4df7-8cea-2cc464744f78-metrics-client-ca\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:21.978970 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:21.978941    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:22.030639 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.030573    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15c4e760-06e6-4df7-8cea-2cc464744f78-sys\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.030639 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.030619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15c4e760-06e6-4df7-8cea-2cc464744f78-root\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.030779 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.030642    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-tls\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.030779 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.030666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-textfile\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.030779 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.030680    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15c4e760-06e6-4df7-8cea-2cc464744f78-sys\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.030779 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.030692    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c4e760-06e6-4df7-8cea-2cc464744f78-metrics-client-ca\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.030779 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.030705    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15c4e760-06e6-4df7-8cea-2cc464744f78-root\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.030779 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.030724    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvzs2\" (UniqueName: \"kubernetes.io/projected/15c4e760-06e6-4df7-8cea-2cc464744f78-kube-api-access-lvzs2\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.030779 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:50:22.030752    2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 04:50:22.031001 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.030774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-accelerators-collector-config\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.031428 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.031390    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-textfile\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.031770 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:50:22.031711    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-tls podName:15c4e760-06e6-4df7-8cea-2cc464744f78 nodeName:}" failed. No retries permitted until 2026-09-16 04:50:22.530800442 +0000 UTC m=+86.196730013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-tls") pod "node-exporter-dqswl" (UID: "15c4e760-06e6-4df7-8cea-2cc464744f78") : secret "node-exporter-tls" not found
Sep 16 04:50:22.032119 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.032093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-wtmp\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.032337 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.032280    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.032586 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.032193    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-accelerators-collector-config\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.032786 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.032394    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-wtmp\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.032923 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.032172    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c4e760-06e6-4df7-8cea-2cc464744f78-metrics-client-ca\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.036473 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.035523    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.039723 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.039697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvzs2\" (UniqueName: \"kubernetes.io/projected/15c4e760-06e6-4df7-8cea-2cc464744f78-kube-api-access-lvzs2\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.084688 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.084665    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:22.193530 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.193505    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:22.353926 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.353854    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:22.488246 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.488225    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:22.535894 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.535873    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-tls\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.538025 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.538000    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15c4e760-06e6-4df7-8cea-2cc464744f78-node-exporter-tls\") pod \"node-exporter-dqswl\" (UID: \"15c4e760-06e6-4df7-8cea-2cc464744f78\") " pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.690533 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.690509    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:22.709629 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.709610    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dqswl"
Sep 16 04:50:22.720652 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:50:22.720631    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c4e760_06e6_4df7_8cea_2cc464744f78.slice/crio-34a5295c2a0cb6c59d4e1066e10eea12b58ad8727c611966fb34071d6a8f2bb4 WatchSource:0}: Error finding container 34a5295c2a0cb6c59d4e1066e10eea12b58ad8727c611966fb34071d6a8f2bb4: Status 404 returned error can't find the container with id 34a5295c2a0cb6c59d4e1066e10eea12b58ad8727c611966fb34071d6a8f2bb4
Sep 16 04:50:22.770564 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.770543    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:22.891319 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:22.891295    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:23.189813 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:23.189786    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqswl" event={"ID":"15c4e760-06e6-4df7-8cea-2cc464744f78","Type":"ContainerStarted","Data":"34a5295c2a0cb6c59d4e1066e10eea12b58ad8727c611966fb34071d6a8f2bb4"}
Sep 16 04:50:24.195317 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:24.195284    2573 generic.go:356] "Generic (PLEG): container finished" podID="15c4e760-06e6-4df7-8cea-2cc464744f78" containerID="c0f60c22a2760adfa67a1d96daa5c9a8011cfb9c47a63c18f8fc9a7dd465d239" exitCode=0
Sep 16 04:50:24.195676 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:24.195333    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqswl" event={"ID":"15c4e760-06e6-4df7-8cea-2cc464744f78","Type":"ContainerDied","Data":"c0f60c22a2760adfa67a1d96daa5c9a8011cfb9c47a63c18f8fc9a7dd465d239"}
Sep 16 04:50:24.635896 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:24.635871    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:24.735831 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:24.735799    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:24.836294 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:24.836262    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:24.937258 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:24.937192    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:25.037906 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:25.037879    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:25.137736 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:25.137712    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:25.202100 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:25.202033    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqswl" event={"ID":"15c4e760-06e6-4df7-8cea-2cc464744f78","Type":"ContainerStarted","Data":"c2fc088f05f9f8b2034fc937e816df73cdaf2f8336af2e723c8ec7e300d5e6e8"}
Sep 16 04:50:25.202415 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:25.202102    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqswl" event={"ID":"15c4e760-06e6-4df7-8cea-2cc464744f78","Type":"ContainerStarted","Data":"1ed00ae4280dfef521648c4d1bbd9669863efe85c94475c84fd0044f39023b2f"}
Sep 16 04:50:25.221141 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:25.221101    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dqswl" podStartSLOduration=3.331476219 podStartE2EDuration="4.221086558s" totalImagesPullingTime="830.705945ms" totalInitContainerRuntime="58.904394ms" isStatefulPod=true podCreationTimestamp="2026-09-16 04:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:25.219506373 +0000 UTC m=+88.885435983" watchObservedRunningTime="2026-09-16 04:50:25.221086558 +0000 UTC m=+88.887016212"
Sep 16 04:50:25.462910 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:25.462852    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:25.521581 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:25.521552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:26.495029 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:26.495000    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:26.561963 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:26.561932    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:26.677509 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:26.677480    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:27.128831 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:27.128804    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:50:28.069613 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:28.069576    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:28.345417 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:28.345351    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:28.446803 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:28.446776    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:30.292933 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:30.292894    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:30.391618 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:30.391575    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:32.137344 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.137309    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:32.142889 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.142850    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l" podUID="102f3bea-c75e-4dea-9234-83c4785a68fc" containerName="registry" containerID="cri-o://543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9" gracePeriod=30
Sep 16 04:50:32.237396 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.237369    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:32.379446 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.379420    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:50:32.506404 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.506337    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-bound-sa-token\") pod \"102f3bea-c75e-4dea-9234-83c4785a68fc\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") "
Sep 16 04:50:32.506404 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.506377    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-trusted-ca\") pod \"102f3bea-c75e-4dea-9234-83c4785a68fc\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") "
Sep 16 04:50:32.506404 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.506394    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") pod \"102f3bea-c75e-4dea-9234-83c4785a68fc\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") "
Sep 16 04:50:32.506637 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.506421    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-image-registry-private-configuration\") pod \"102f3bea-c75e-4dea-9234-83c4785a68fc\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") "
Sep 16 04:50:32.506637 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.506587    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/102f3bea-c75e-4dea-9234-83c4785a68fc-ca-trust-extracted\") pod \"102f3bea-c75e-4dea-9234-83c4785a68fc\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") "
Sep 16 04:50:32.506739 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.506703    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnntg\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-kube-api-access-fnntg\") pod \"102f3bea-c75e-4dea-9234-83c4785a68fc\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") "
Sep 16 04:50:32.506739 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.506732    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-certificates\") pod \"102f3bea-c75e-4dea-9234-83c4785a68fc\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") "
Sep 16 04:50:32.506835 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.506758    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-installation-pull-secrets\") pod \"102f3bea-c75e-4dea-9234-83c4785a68fc\" (UID: \"102f3bea-c75e-4dea-9234-83c4785a68fc\") "
Sep 16 04:50:32.506880 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.506848    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "102f3bea-c75e-4dea-9234-83c4785a68fc" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 04:50:32.507006 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.506966    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-trusted-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:50:32.507314 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.507276    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "102f3bea-c75e-4dea-9234-83c4785a68fc" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 04:50:32.508986 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.508940    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "102f3bea-c75e-4dea-9234-83c4785a68fc" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 04:50:32.509203 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.509175    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "102f3bea-c75e-4dea-9234-83c4785a68fc" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 04:50:32.509329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.509226    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "102f3bea-c75e-4dea-9234-83c4785a68fc" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 04:50:32.509329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.509266    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "102f3bea-c75e-4dea-9234-83c4785a68fc" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 04:50:32.509329 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.509305    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-kube-api-access-fnntg" (OuterVolumeSpecName: "kube-api-access-fnntg") pod "102f3bea-c75e-4dea-9234-83c4785a68fc" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc"). InnerVolumeSpecName "kube-api-access-fnntg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 04:50:32.515260 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.515229    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102f3bea-c75e-4dea-9234-83c4785a68fc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "102f3bea-c75e-4dea-9234-83c4785a68fc" (UID: "102f3bea-c75e-4dea-9234-83c4785a68fc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 04:50:32.607301 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.607276    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-image-registry-private-configuration\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:50:32.607301 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.607298    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/102f3bea-c75e-4dea-9234-83c4785a68fc-ca-trust-extracted\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:50:32.607427 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.607309    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnntg\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-kube-api-access-fnntg\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:50:32.607427 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.607318    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-certificates\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:50:32.607427 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.607327    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/102f3bea-c75e-4dea-9234-83c4785a68fc-installation-pull-secrets\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:50:32.607427 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.607336    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-bound-sa-token\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:50:32.607427 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:32.607344    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/102f3bea-c75e-4dea-9234-83c4785a68fc-registry-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:50:33.231732 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:33.231696    2573 generic.go:356] "Generic (PLEG): container finished" podID="102f3bea-c75e-4dea-9234-83c4785a68fc" containerID="543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9" exitCode=0
Sep 16 04:50:33.232099 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:33.231751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l" event={"ID":"102f3bea-c75e-4dea-9234-83c4785a68fc","Type":"ContainerDied","Data":"543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9"}
Sep 16 04:50:33.232099 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:33.231791    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l" event={"ID":"102f3bea-c75e-4dea-9234-83c4785a68fc","Type":"ContainerDied","Data":"65bd43ed4e8dcc30678ae1010ef63c24c6ed8b5bbcbe25bea6a909dc6624f1ec"}
Sep 16 04:50:33.232099 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:33.231813    2573 scope.go:117] "RemoveContainer" containerID="543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9"
Sep 16 04:50:33.232099 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:33.231754    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675cb69c4f-hx57l"
Sep 16 04:50:33.241350 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:33.241329    2573 scope.go:117] "RemoveContainer" containerID="543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9"
Sep 16 04:50:33.241690 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:50:33.241664    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9\": container with ID starting with 543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9 not found: ID does not exist" containerID="543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9"
Sep 16 04:50:33.241802 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:33.241694    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9"} err="failed to get container status \"543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9\": rpc error: code = NotFound desc = could not find container \"543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9\": container with ID starting with 543245b939f104eb339474baa6ec1252a5b159080fdeb064ba8e21adae829ec9 not found: ID does not exist"
Sep 16 04:50:33.246850 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:33.246825    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675cb69c4f-hx57l"]
Sep 16 04:50:33.252385 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:33.252350    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-675cb69c4f-hx57l"]
Sep 16 04:50:34.785185 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:34.785156    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102f3bea-c75e-4dea-9234-83c4785a68fc" path="/var/lib/kubelet/pods/102f3bea-c75e-4dea-9234-83c4785a68fc/volumes"
Sep 16 04:50:39.140818 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:39.140789    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-64c4f"
Sep 16 04:50:39.349035 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:39.349004    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:39.450222 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:39.450151    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:39.556082 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:39.556055    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:39.658962 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:39.658936    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:40.884456 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:40.884426    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:40.985058 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:40.985032    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:41.085963 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:41.085940    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:41.489387 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:41.489364    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:41.593489 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:41.593464    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:41.870189 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:41.870133    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:41.972315 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:41.972292    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:42.211176 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:42.211145    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:42.270095 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:42.270072    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:57.206008 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:57.205972    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:50:57.259950 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:50:57.259920    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:51:04.347192 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:51:04.347152    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:51:04.447554 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:51:04.447522    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:51:04.500526 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:51:04.500497    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:51:40.980479 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:51:40.980439    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:51:41.182323 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:51:41.182291    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:51:50.251888 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:51:50.251795    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:51:50.395412 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:51:50.395381    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:51:50.497147 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:51:50.497116    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:51:50.990669 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:51:50.990639    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:52:01.304783 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:52:01.304747    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:52:01.404470 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:52:01.404441    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:52:01.469450 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:52:01.469425    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:52:03.613426 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:52:03.613390    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:52:56.212926 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:52:56.212892    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:52:56.286964 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:52:56.286933    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:07.653978 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:07.653942    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:07.714044 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:07.714015    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:08.172857 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:08.172831    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:08.273800 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:08.273774    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:12.217203 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:12.217160    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:12.317989 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:12.317953    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:18.633831 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:18.633793    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:18.733865 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:18.733832    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:18.810064 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:18.810029    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:28.240652 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:28.240540    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:53:56.689522 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:56.689483    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 04:53:56.694981 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:56.694955    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 04:53:56.696195 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:53:56.696179    2573 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 04:54:14.644504 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:54:14.644462    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:54:31.036707 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:54:31.036672    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:54:31.127916 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:54:31.127867    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:54:57.797242 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:54:57.797159    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:54:58.645050 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:54:58.645007    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:01.468036 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:01.467995    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:01.565285 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:01.565253    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:03.422385 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:03.422350    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:03.676996 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:03.676895    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:03.809540 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:03.809515    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:05.961304 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:05.961275    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:06.150758 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:06.150716    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:08.235880 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:08.235840    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:08.306434 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:08.306402    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:11.930118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:11.930089    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:13.584957 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:13.584919    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:14.617699 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:14.617657    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:14.720559 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:14.720530    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:16.631924 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.631893    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xkfrl"]
Sep 16 04:55:16.632323 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.632144    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102f3bea-c75e-4dea-9234-83c4785a68fc" containerName="registry"
Sep 16 04:55:16.632323 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.632156    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="102f3bea-c75e-4dea-9234-83c4785a68fc" containerName="registry"
Sep 16 04:55:16.632323 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.632212    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="102f3bea-c75e-4dea-9234-83c4785a68fc" containerName="registry"
Sep 16 04:55:16.634319 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.634293    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xkfrl"
Sep 16 04:55:16.634614 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.634568    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"]
Sep 16 04:55:16.636439 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.636415    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 04:55:16.636536 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.636491    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"
Sep 16 04:55:16.636945 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.636924    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qm2dl\""
Sep 16 04:55:16.637050 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.636926    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 04:55:16.638389 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.638369    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 04:55:16.638495 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.638400    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-67rkr\""
Sep 16 04:55:16.638495 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.638418    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 04:55:16.643613 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.643576    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xkfrl"]
Sep 16 04:55:16.646314 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.646289    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"]
Sep 16 04:55:16.750556 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.750529    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ddc9a3d-f768-4151-8685-7db82c15e9b9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-twxxl\" (UID: \"1ddc9a3d-f768-4151-8685-7db82c15e9b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"
Sep 16 04:55:16.750556 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.750557    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xkxd\" (UniqueName: \"kubernetes.io/projected/1ddc9a3d-f768-4151-8685-7db82c15e9b9-kube-api-access-4xkxd\") pod \"openshift-lws-operator-fd8ccff4c-twxxl\" (UID: \"1ddc9a3d-f768-4151-8685-7db82c15e9b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"
Sep 16 04:55:16.750727 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.750587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48f8657f-fae6-413a-846e-c29eba14ac64-bound-sa-token\") pod \"cert-manager-55b4d954d5-xkfrl\" (UID: \"48f8657f-fae6-413a-846e-c29eba14ac64\") " pod="cert-manager/cert-manager-55b4d954d5-xkfrl"
Sep 16 04:55:16.750727 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.750622    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvbl7\" (UniqueName: \"kubernetes.io/projected/48f8657f-fae6-413a-846e-c29eba14ac64-kube-api-access-mvbl7\") pod \"cert-manager-55b4d954d5-xkfrl\" (UID: \"48f8657f-fae6-413a-846e-c29eba14ac64\") " pod="cert-manager/cert-manager-55b4d954d5-xkfrl"
Sep 16 04:55:16.851207 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.851182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbl7\" (UniqueName: \"kubernetes.io/projected/48f8657f-fae6-413a-846e-c29eba14ac64-kube-api-access-mvbl7\") pod \"cert-manager-55b4d954d5-xkfrl\" (UID: \"48f8657f-fae6-413a-846e-c29eba14ac64\") " pod="cert-manager/cert-manager-55b4d954d5-xkfrl"
Sep 16 04:55:16.851306 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.851239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ddc9a3d-f768-4151-8685-7db82c15e9b9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-twxxl\" (UID: \"1ddc9a3d-f768-4151-8685-7db82c15e9b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"
Sep 16 04:55:16.851306 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.851265    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xkxd\" (UniqueName: \"kubernetes.io/projected/1ddc9a3d-f768-4151-8685-7db82c15e9b9-kube-api-access-4xkxd\") pod \"openshift-lws-operator-fd8ccff4c-twxxl\" (UID: \"1ddc9a3d-f768-4151-8685-7db82c15e9b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"
Sep 16 04:55:16.851395 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.851375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48f8657f-fae6-413a-846e-c29eba14ac64-bound-sa-token\") pod \"cert-manager-55b4d954d5-xkfrl\" (UID: \"48f8657f-fae6-413a-846e-c29eba14ac64\") " pod="cert-manager/cert-manager-55b4d954d5-xkfrl"
Sep 16 04:55:16.851730 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.851710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ddc9a3d-f768-4151-8685-7db82c15e9b9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-twxxl\" (UID: \"1ddc9a3d-f768-4151-8685-7db82c15e9b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"
Sep 16 04:55:16.859451 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.859420    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48f8657f-fae6-413a-846e-c29eba14ac64-bound-sa-token\") pod \"cert-manager-55b4d954d5-xkfrl\" (UID: \"48f8657f-fae6-413a-846e-c29eba14ac64\") " pod="cert-manager/cert-manager-55b4d954d5-xkfrl"
Sep 16 04:55:16.859537 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.859484    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xkxd\" (UniqueName: \"kubernetes.io/projected/1ddc9a3d-f768-4151-8685-7db82c15e9b9-kube-api-access-4xkxd\") pod \"openshift-lws-operator-fd8ccff4c-twxxl\" (UID: \"1ddc9a3d-f768-4151-8685-7db82c15e9b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"
Sep 16 04:55:16.859577 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.859533    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbl7\" (UniqueName: \"kubernetes.io/projected/48f8657f-fae6-413a-846e-c29eba14ac64-kube-api-access-mvbl7\") pod \"cert-manager-55b4d954d5-xkfrl\" (UID: \"48f8657f-fae6-413a-846e-c29eba14ac64\") " pod="cert-manager/cert-manager-55b4d954d5-xkfrl"
Sep 16 04:55:16.945469 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.945402    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xkfrl"
Sep 16 04:55:16.951486 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:16.951457    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"
Sep 16 04:55:17.218903 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:17.218829    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xkfrl"]
Sep 16 04:55:17.222453 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:55:17.222421    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48f8657f_fae6_413a_846e_c29eba14ac64.slice/crio-c2952eb47fcabbfd785ec834f3d943c7093a2f0050363e8e9044eb0a7f4c8e62 WatchSource:0}: Error finding container c2952eb47fcabbfd785ec834f3d943c7093a2f0050363e8e9044eb0a7f4c8e62: Status 404 returned error can't find the container with id c2952eb47fcabbfd785ec834f3d943c7093a2f0050363e8e9044eb0a7f4c8e62
Sep 16 04:55:17.222725 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:17.222695    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl"]
Sep 16 04:55:17.224451 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:17.224431    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 04:55:17.225964 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:55:17.225944    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ddc9a3d_f768_4151_8685_7db82c15e9b9.slice/crio-ab9289ee03182b8978342bb22ff1b5fa3144c81dea0d61661d2c9ba3a87dfd9a WatchSource:0}: Error finding container ab9289ee03182b8978342bb22ff1b5fa3144c81dea0d61661d2c9ba3a87dfd9a: Status 404 returned error can't find the container with id ab9289ee03182b8978342bb22ff1b5fa3144c81dea0d61661d2c9ba3a87dfd9a
Sep 16 04:55:18.165548 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:18.165510    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xkfrl" event={"ID":"48f8657f-fae6-413a-846e-c29eba14ac64","Type":"ContainerStarted","Data":"c2952eb47fcabbfd785ec834f3d943c7093a2f0050363e8e9044eb0a7f4c8e62"}
Sep 16 04:55:18.167577 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:18.167546    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl" event={"ID":"1ddc9a3d-f768-4151-8685-7db82c15e9b9","Type":"ContainerStarted","Data":"ab9289ee03182b8978342bb22ff1b5fa3144c81dea0d61661d2c9ba3a87dfd9a"}
Sep 16 04:55:22.184694 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:22.184654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xkfrl" event={"ID":"48f8657f-fae6-413a-846e-c29eba14ac64","Type":"ContainerStarted","Data":"e124065ffd687ae090845b163934a93c8fa7ebe576902e168d6fdedf91742fcb"}
Sep 16 04:55:22.186291 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:22.186270    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl" event={"ID":"1ddc9a3d-f768-4151-8685-7db82c15e9b9","Type":"ContainerStarted","Data":"5643b1e0571a95872e677405c643cdc0b89e19e563b73995cd0cd8ca7d1af20c"}
Sep 16 04:55:22.205847 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:22.205799    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xkfrl" podStartSLOduration=2.08841868 podStartE2EDuration="6.20578597s" totalImagesPullingTime="4.11736729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:55:22.204546205 +0000 UTC m=+385.870475796" watchObservedRunningTime="2026-09-16 04:55:22.20578597 +0000 UTC m=+385.871715562"
Sep 16 04:55:22.218719 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:22.218655    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twxxl" podStartSLOduration=2.13379813 podStartE2EDuration="6.218619913s" totalImagesPullingTime="4.084821783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:55:22.218322892 +0000 UTC m=+385.884252485" watchObservedRunningTime="2026-09-16 04:55:22.218619913 +0000 UTC m=+385.884549498"
Sep 16 04:55:22.984246 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:22.984210    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:23.085483 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:23.085452    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:24.948915 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:24.948886    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:35.598301 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:35.598260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:36.824075 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:36.824030    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:36.926079 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:36.926052    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:37.378016 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.377992    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:37.685416 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.685356    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:37.787278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.787252    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:37.825566 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.825544    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"]
Sep 16 04:55:37.830360 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.830344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:37.832363 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.832339    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 04:55:37.832475 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.832394    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 04:55:37.832544 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.832528    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zn4sz\""
Sep 16 04:55:37.832650 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.832634    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 04:55:37.832703 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.832666    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 04:55:37.847079 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.847061    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"]
Sep 16 04:55:37.944426 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.944376    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:37.987724 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.987701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8661495-0d52-469e-b831-f10980137a67-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:37.987784 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.987736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8661495-0d52-469e-b831-f10980137a67-tmp\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:37.987784 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.987764    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk8gw\" (UniqueName: \"kubernetes.io/projected/b8661495-0d52-469e-b831-f10980137a67-kube-api-access-gk8gw\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:37.987850 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.987798    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8661495-0d52-469e-b831-f10980137a67-manifests\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:37.987850 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:37.987815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8661495-0d52-469e-b831-f10980137a67-webhook-cert\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.045728 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.045703    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:38.089029 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.089002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8661495-0d52-469e-b831-f10980137a67-tmp\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.089134 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.089046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8gw\" (UniqueName: \"kubernetes.io/projected/b8661495-0d52-469e-b831-f10980137a67-kube-api-access-gk8gw\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.089134 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.089076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8661495-0d52-469e-b831-f10980137a67-manifests\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.089134 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.089101    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8661495-0d52-469e-b831-f10980137a67-webhook-cert\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.089283 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.089150    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8661495-0d52-469e-b831-f10980137a67-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.089505 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.089473    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8661495-0d52-469e-b831-f10980137a67-manifests\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.091560 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.091538    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8661495-0d52-469e-b831-f10980137a67-tmp\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.091886 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.091866    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8661495-0d52-469e-b831-f10980137a67-webhook-cert\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.092357 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.092325    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8661495-0d52-469e-b831-f10980137a67-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.097084 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.097059    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk8gw\" (UniqueName: \"kubernetes.io/projected/b8661495-0d52-469e-b831-f10980137a67-kube-api-access-gk8gw\") pod \"opendatahub-operator-controller-manager-756589c867-w899b\" (UID: \"b8661495-0d52-469e-b831-f10980137a67\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.139881 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.139857    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:38.144063 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.144046    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:38.216462 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.216404    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:38.320802 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.320780    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:38.443350 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:38.441125    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"]
Sep 16 04:55:38.445022 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:55:38.444987    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8661495_0d52_469e_b831_f10980137a67.slice/crio-1091e6b4ff138f2f1f3bf669414f2a5bca31d71ba7e6a0b29566c5f18b038088 WatchSource:0}: Error finding container 1091e6b4ff138f2f1f3bf669414f2a5bca31d71ba7e6a0b29566c5f18b038088: Status 404 returned error can't find the container with id 1091e6b4ff138f2f1f3bf669414f2a5bca31d71ba7e6a0b29566c5f18b038088
Sep 16 04:55:39.254391 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:39.254347    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b" event={"ID":"b8661495-0d52-469e-b831-f10980137a67","Type":"ContainerStarted","Data":"1091e6b4ff138f2f1f3bf669414f2a5bca31d71ba7e6a0b29566c5f18b038088"}
Sep 16 04:55:42.266751 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:42.266722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b" event={"ID":"b8661495-0d52-469e-b831-f10980137a67","Type":"ContainerStarted","Data":"58b41fe07e496a93bf6480a28eeb507df5bc0ee7163f9a83516e18e135e1511e"}
Sep 16 04:55:43.271412 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:43.271378    2573 generic.go:356] "Generic (PLEG): container finished" podID="b8661495-0d52-469e-b831-f10980137a67" containerID="58b41fe07e496a93bf6480a28eeb507df5bc0ee7163f9a83516e18e135e1511e" exitCode=0
Sep 16 04:55:43.271897 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:43.271427    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b" event={"ID":"b8661495-0d52-469e-b831-f10980137a67","Type":"ContainerDied","Data":"58b41fe07e496a93bf6480a28eeb507df5bc0ee7163f9a83516e18e135e1511e"}
Sep 16 04:55:44.279489 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:44.279460    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b" event={"ID":"b8661495-0d52-469e-b831-f10980137a67","Type":"ContainerStarted","Data":"f117c10db0994fb526a619fda722457c380853c6a6bcf113e09d9e39c67b3f13"}
Sep 16 04:55:44.279847 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:44.279585    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:44.302666 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:44.302623    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b" podStartSLOduration=2.717932176 podStartE2EDuration="7.302585329s" totalImagesPullingTime="3.566651201s" totalInitContainerRuntime="1.018001952s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:55:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:55:44.301897185 +0000 UTC m=+407.967826795" watchObservedRunningTime="2026-09-16 04:55:44.302585329 +0000 UTC m=+407.968514920"
Sep 16 04:55:46.324012 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:46.323977    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:46.373481 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:46.373447    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:46.433282 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:46.433245    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:47.687424 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:47.687393    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:47.755882 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:47.755855    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:55.285107 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:55.285083    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-w899b"
Sep 16 04:55:57.722141 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:57.722106    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:57.852685 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:57.852650    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:58.202831 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.202804    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:58.217028 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.217003    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"]
Sep 16 04:55:58.220010 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.219995    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.224516 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.224494    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 04:55:58.224656 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.224638    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 04:55:58.224739 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.224669    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 04:55:58.224795 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.224736    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 04:55:58.236586 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.236560    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"]
Sep 16 04:55:58.322461 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.322438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd-tls-certs\") pod \"kube-auth-proxy-6965d5747b-dsn8t\" (UID: \"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.322554 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.322481    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd-tmp\") pod \"kube-auth-proxy-6965d5747b-dsn8t\" (UID: \"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.322554 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.322499    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wms\" (UniqueName: \"kubernetes.io/projected/dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd-kube-api-access-q5wms\") pod \"kube-auth-proxy-6965d5747b-dsn8t\" (UID: \"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.423175 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.423152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd-tls-certs\") pod \"kube-auth-proxy-6965d5747b-dsn8t\" (UID: \"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.423278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.423200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd-tmp\") pod \"kube-auth-proxy-6965d5747b-dsn8t\" (UID: \"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.423278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.423220    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wms\" (UniqueName: \"kubernetes.io/projected/dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd-kube-api-access-q5wms\") pod \"kube-auth-proxy-6965d5747b-dsn8t\" (UID: \"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.425704 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.425683    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd-tmp\") pod \"kube-auth-proxy-6965d5747b-dsn8t\" (UID: \"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.425872 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.425855    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd-tls-certs\") pod \"kube-auth-proxy-6965d5747b-dsn8t\" (UID: \"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.433163 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.433137    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wms\" (UniqueName: \"kubernetes.io/projected/dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd-kube-api-access-q5wms\") pod \"kube-auth-proxy-6965d5747b-dsn8t\" (UID: \"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.434400 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.434375    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:58.529189 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.529127    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"
Sep 16 04:55:58.817055 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:58.816982    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t"]
Sep 16 04:55:58.821915 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:55:58.821879    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa1784a_f1bd_4c5c_8dae_3dc8fcdb0dfd.slice/crio-cf77f4bdcf5fcbbcd3ca03693f9cb3e9892b5cd43832653e33a8df96e384d127 WatchSource:0}: Error finding container cf77f4bdcf5fcbbcd3ca03693f9cb3e9892b5cd43832653e33a8df96e384d127: Status 404 returned error can't find the container with id cf77f4bdcf5fcbbcd3ca03693f9cb3e9892b5cd43832653e33a8df96e384d127
Sep 16 04:55:59.333992 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:59.333961    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t" event={"ID":"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd","Type":"ContainerStarted","Data":"cf77f4bdcf5fcbbcd3ca03693f9cb3e9892b5cd43832653e33a8df96e384d127"}
Sep 16 04:55:59.536318 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:59.536283    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:55:59.645031 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:55:59.645005    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:00.553775 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:00.553745    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:02.352869 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:02.352830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t" event={"ID":"dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd","Type":"ContainerStarted","Data":"69001536f54439e69558abb0abce8bc55281eedac37263b3b8c1c1c08cc0e812"}
Sep 16 04:56:02.372189 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:02.372142    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6965d5747b-dsn8t" podStartSLOduration=1.598065626 podStartE2EDuration="4.372127685s" totalImagesPullingTime="2.774062059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:56:02.370703176 +0000 UTC m=+426.036632783" watchObservedRunningTime="2026-09-16 04:56:02.372127685 +0000 UTC m=+426.038057277"
Sep 16 04:56:04.074757 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:04.074712    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:04.176792 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:04.176764    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:04.669853 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:04.669827    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:04.769831 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:04.769803    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:04.890972 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:04.890945    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:06.909869 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:06.909827    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:06.966483 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:06.966450    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:08.758029 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.757994    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lxpdd"]
Sep 16 04:56:08.760415 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.760394    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:08.763097 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.763071    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6qqw5\""
Sep 16 04:56:08.763401 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.763383    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 16 04:56:08.772901 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.772873    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lxpdd"]
Sep 16 04:56:08.784139 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.784115    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:08.800297 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.800266    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdl4h\" (UniqueName: \"kubernetes.io/projected/532d8a7e-b308-41f0-8779-351326953242-kube-api-access-wdl4h\") pod \"odh-model-controller-7998c67dc4-lxpdd\" (UID: \"532d8a7e-b308-41f0-8779-351326953242\") " pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:08.800407 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.800328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/532d8a7e-b308-41f0-8779-351326953242-cert\") pod \"odh-model-controller-7998c67dc4-lxpdd\" (UID: \"532d8a7e-b308-41f0-8779-351326953242\") " pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:08.879892 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.879866    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:08.901052 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.901033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/532d8a7e-b308-41f0-8779-351326953242-cert\") pod \"odh-model-controller-7998c67dc4-lxpdd\" (UID: \"532d8a7e-b308-41f0-8779-351326953242\") " pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:08.901123 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.901076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdl4h\" (UniqueName: \"kubernetes.io/projected/532d8a7e-b308-41f0-8779-351326953242-kube-api-access-wdl4h\") pod \"odh-model-controller-7998c67dc4-lxpdd\" (UID: \"532d8a7e-b308-41f0-8779-351326953242\") " pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:08.901205 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:56:08.901188    2573 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 04:56:08.901279 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:56:08.901269    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/532d8a7e-b308-41f0-8779-351326953242-cert podName:532d8a7e-b308-41f0-8779-351326953242 nodeName:}" failed. No retries permitted until 2026-09-16 04:56:09.401249175 +0000 UTC m=+433.067178767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/532d8a7e-b308-41f0-8779-351326953242-cert") pod "odh-model-controller-7998c67dc4-lxpdd" (UID: "532d8a7e-b308-41f0-8779-351326953242") : secret "odh-model-controller-webhook-cert" not found
Sep 16 04:56:08.912930 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.912902    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdl4h\" (UniqueName: \"kubernetes.io/projected/532d8a7e-b308-41f0-8779-351326953242-kube-api-access-wdl4h\") pod \"odh-model-controller-7998c67dc4-lxpdd\" (UID: \"532d8a7e-b308-41f0-8779-351326953242\") " pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:08.982886 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:08.982862    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:09.080030 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.079968    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:09.371943 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.371883    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-c8zqs"]
Sep 16 04:56:09.374152 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.374136    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:56:09.377158 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.377133    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 16 04:56:09.385219 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.385197    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-c8zqs"]
Sep 16 04:56:09.388488 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.388466    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:09.405444 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.405416    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2nw9\" (UniqueName: \"kubernetes.io/projected/2339d094-fc27-46a3-bd1a-8b3b738dcbd2-kube-api-access-l2nw9\") pod \"model-serving-api-54d89bd5fc-c8zqs\" (UID: \"2339d094-fc27-46a3-bd1a-8b3b738dcbd2\") " pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:56:09.405536 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.405469    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/532d8a7e-b308-41f0-8779-351326953242-cert\") pod \"odh-model-controller-7998c67dc4-lxpdd\" (UID: \"532d8a7e-b308-41f0-8779-351326953242\") " pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:09.405536 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.405524    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2339d094-fc27-46a3-bd1a-8b3b738dcbd2-tls-certs\") pod \"model-serving-api-54d89bd5fc-c8zqs\" (UID: \"2339d094-fc27-46a3-bd1a-8b3b738dcbd2\") " pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:56:09.407719 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.407702    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/532d8a7e-b308-41f0-8779-351326953242-cert\") pod \"odh-model-controller-7998c67dc4-lxpdd\" (UID: \"532d8a7e-b308-41f0-8779-351326953242\") " pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:09.506392 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.506371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2nw9\" (UniqueName: \"kubernetes.io/projected/2339d094-fc27-46a3-bd1a-8b3b738dcbd2-kube-api-access-l2nw9\") pod \"model-serving-api-54d89bd5fc-c8zqs\" (UID: \"2339d094-fc27-46a3-bd1a-8b3b738dcbd2\") " pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:56:09.506479 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.506420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2339d094-fc27-46a3-bd1a-8b3b738dcbd2-tls-certs\") pod \"model-serving-api-54d89bd5fc-c8zqs\" (UID: \"2339d094-fc27-46a3-bd1a-8b3b738dcbd2\") " pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:56:09.506522 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:56:09.506498    2573 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 04:56:09.506559 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:56:09.506550    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2339d094-fc27-46a3-bd1a-8b3b738dcbd2-tls-certs podName:2339d094-fc27-46a3-bd1a-8b3b738dcbd2 nodeName:}" failed. No retries permitted until 2026-09-16 04:56:10.006537006 +0000 UTC m=+433.672466576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2339d094-fc27-46a3-bd1a-8b3b738dcbd2-tls-certs") pod "model-serving-api-54d89bd5fc-c8zqs" (UID: "2339d094-fc27-46a3-bd1a-8b3b738dcbd2") : secret "model-serving-api-tls" not found
Sep 16 04:56:09.514843 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.514823    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2nw9\" (UniqueName: \"kubernetes.io/projected/2339d094-fc27-46a3-bd1a-8b3b738dcbd2-kube-api-access-l2nw9\") pod \"model-serving-api-54d89bd5fc-c8zqs\" (UID: \"2339d094-fc27-46a3-bd1a-8b3b738dcbd2\") " pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:56:09.673549 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.673516    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:09.750357 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.750332    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:09.942450 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:09.942389    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lxpdd"]
Sep 16 04:56:09.945937 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:56:09.945912    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532d8a7e_b308_41f0_8779_351326953242.slice/crio-d853a0ea58c6037b9a6dedd2831e1075b43dc300faeec01afc017f5d4233c8d5 WatchSource:0}: Error finding container d853a0ea58c6037b9a6dedd2831e1075b43dc300faeec01afc017f5d4233c8d5: Status 404 returned error can't find the container with id d853a0ea58c6037b9a6dedd2831e1075b43dc300faeec01afc017f5d4233c8d5
Sep 16 04:56:10.011165 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:10.011144    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2339d094-fc27-46a3-bd1a-8b3b738dcbd2-tls-certs\") pod \"model-serving-api-54d89bd5fc-c8zqs\" (UID: \"2339d094-fc27-46a3-bd1a-8b3b738dcbd2\") " pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:56:10.013477 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:10.013446    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2339d094-fc27-46a3-bd1a-8b3b738dcbd2-tls-certs\") pod \"model-serving-api-54d89bd5fc-c8zqs\" (UID: \"2339d094-fc27-46a3-bd1a-8b3b738dcbd2\") " pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:56:10.284880 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:10.284824    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:56:10.382296 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:10.382259    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd" event={"ID":"532d8a7e-b308-41f0-8779-351326953242","Type":"ContainerStarted","Data":"d853a0ea58c6037b9a6dedd2831e1075b43dc300faeec01afc017f5d4233c8d5"}
Sep 16 04:56:10.557208 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:10.557185    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-c8zqs"]
Sep 16 04:56:10.559661 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:56:10.559637    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2339d094_fc27_46a3_bd1a_8b3b738dcbd2.slice/crio-73ba5e4fbd64f1e6ae1ef37f5dc622746d770b9f9da2873e02eaeedd81cdf015 WatchSource:0}: Error finding container 73ba5e4fbd64f1e6ae1ef37f5dc622746d770b9f9da2873e02eaeedd81cdf015: Status 404 returned error can't find the container with id 73ba5e4fbd64f1e6ae1ef37f5dc622746d770b9f9da2873e02eaeedd81cdf015
Sep 16 04:56:11.389278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:11.389047    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs" event={"ID":"2339d094-fc27-46a3-bd1a-8b3b738dcbd2","Type":"ContainerStarted","Data":"73ba5e4fbd64f1e6ae1ef37f5dc622746d770b9f9da2873e02eaeedd81cdf015"}
Sep 16 04:56:13.399358 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:13.399321    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd" event={"ID":"532d8a7e-b308-41f0-8779-351326953242","Type":"ContainerStarted","Data":"50d85cf1ee32e5cff609f3468940055d93a4cd664dd7bb7004eb4ef430fa3514"}
Sep 16 04:56:13.399847 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:13.399541    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:13.401053 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:13.401023    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs" event={"ID":"2339d094-fc27-46a3-bd1a-8b3b738dcbd2","Type":"ContainerStarted","Data":"2c683fbc60e6f2dfbda91459b389394ec577cef169084a5f0853589cbec374ef"}
Sep 16 04:56:13.401245 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:13.401229    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:56:13.418127 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:13.418080    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd" podStartSLOduration=2.278860552 podStartE2EDuration="5.418062294s" totalImagesPullingTime="3.139201742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:56:13.417102425 +0000 UTC m=+437.083032049" watchObservedRunningTime="2026-09-16 04:56:13.418062294 +0000 UTC m=+437.083991914"
Sep 16 04:56:13.443877 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:13.443835    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs" podStartSLOduration=1.922465691 podStartE2EDuration="4.443822051s" totalImagesPullingTime="2.52135636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:56:13.442573432 +0000 UTC m=+437.108503057" watchObservedRunningTime="2026-09-16 04:56:13.443822051 +0000 UTC m=+437.109751642"
Sep 16 04:56:15.234229 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.234190    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"]
Sep 16 04:56:15.236536 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.236516    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.239121 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.239096    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 04:56:15.239247 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.239160    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 04:56:15.239247 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.239186    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 04:56:15.239247 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.239203    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vx458\""
Sep 16 04:56:15.239474 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.239456    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 04:56:15.252202 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.252175    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"]
Sep 16 04:56:15.353098 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.353069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d91007-8ea8-4099-96ca-942aafbd5768-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.353198 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.353101    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04d91007-8ea8-4099-96ca-942aafbd5768-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.353198 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.353124    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04d91007-8ea8-4099-96ca-942aafbd5768-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.353198 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.353144    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04d91007-8ea8-4099-96ca-942aafbd5768-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.353312 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.353196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d91007-8ea8-4099-96ca-942aafbd5768-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.353312 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.353227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04d91007-8ea8-4099-96ca-942aafbd5768-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.353312 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.353248    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtgjf\" (UniqueName: \"kubernetes.io/projected/04d91007-8ea8-4099-96ca-942aafbd5768-kube-api-access-jtgjf\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.454029 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.453996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d91007-8ea8-4099-96ca-942aafbd5768-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.454132 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.454033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04d91007-8ea8-4099-96ca-942aafbd5768-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.454132 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.454056    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04d91007-8ea8-4099-96ca-942aafbd5768-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.454132 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.454084    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04d91007-8ea8-4099-96ca-942aafbd5768-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.454132 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.454107    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d91007-8ea8-4099-96ca-942aafbd5768-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.454310 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.454134    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04d91007-8ea8-4099-96ca-942aafbd5768-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.454310 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.454164    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtgjf\" (UniqueName: \"kubernetes.io/projected/04d91007-8ea8-4099-96ca-942aafbd5768-kube-api-access-jtgjf\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.454957 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.454931    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04d91007-8ea8-4099-96ca-942aafbd5768-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.456520 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.456487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d91007-8ea8-4099-96ca-942aafbd5768-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.456857 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.456836    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04d91007-8ea8-4099-96ca-942aafbd5768-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.456993 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.456951    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04d91007-8ea8-4099-96ca-942aafbd5768-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.457029 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.457005    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04d91007-8ea8-4099-96ca-942aafbd5768-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.462317 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.462287    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d91007-8ea8-4099-96ca-942aafbd5768-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.463078 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.463056    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtgjf\" (UniqueName: \"kubernetes.io/projected/04d91007-8ea8-4099-96ca-942aafbd5768-kube-api-access-jtgjf\") pod \"istiod-openshift-gateway-5677ccb49b-lm4ts\" (UID: \"04d91007-8ea8-4099-96ca-942aafbd5768\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.546126 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.546062    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:15.826525 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:15.826461    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"]
Sep 16 04:56:15.829454 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:56:15.829421    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d91007_8ea8_4099_96ca_942aafbd5768.slice/crio-782b56c70eb5d582523d45a53b101433126ab3550075ec995e3bdb5c272e755e WatchSource:0}: Error finding container 782b56c70eb5d582523d45a53b101433126ab3550075ec995e3bdb5c272e755e: Status 404 returned error can't find the container with id 782b56c70eb5d582523d45a53b101433126ab3550075ec995e3bdb5c272e755e
Sep 16 04:56:16.415325 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:16.415294    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts" event={"ID":"04d91007-8ea8-4099-96ca-942aafbd5768","Type":"ContainerStarted","Data":"782b56c70eb5d582523d45a53b101433126ab3550075ec995e3bdb5c272e755e"}
Sep 16 04:56:16.639536 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:16.639479    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:16.770208 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:16.770081    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:18.089655 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:18.089619    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 04:56:18.089894 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:18.089695    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 04:56:18.344322 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:18.344237    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:18.424384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:18.424349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts" event={"ID":"04d91007-8ea8-4099-96ca-942aafbd5768","Type":"ContainerStarted","Data":"153c022436448ffc672769ca8855787a27689b501fe8eba0461896714f00fba9"}
Sep 16 04:56:18.424540 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:18.424490    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:18.476573 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:18.476530    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts" podStartSLOduration=1.218670506 podStartE2EDuration="3.476517328s" totalImagesPullingTime="2.257846822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:56:18.474384035 +0000 UTC m=+442.140313648" watchObservedRunningTime="2026-09-16 04:56:18.476517328 +0000 UTC m=+442.142446920"
Sep 16 04:56:18.513073 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:18.513037    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:18.633803 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:18.633778    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:19.123733 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:19.123647    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:19.429514 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:19.429482    2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-lm4ts container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 04:56:19.429709 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:19.429535    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts" podUID="04d91007-8ea8-4099-96ca-942aafbd5768" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 04:56:19.741399 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:19.741293    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:19.856938 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:19.856909    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:19.921710 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:19.921679    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:22.429280 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:22.429237    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lm4ts"
Sep 16 04:56:22.579576 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:22.579550    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:22.635540 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:22.635515    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:56:24.407320 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:24.407290    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lxpdd"
Sep 16 04:56:24.409169 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:56:24.409147    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-c8zqs"
Sep 16 04:57:09.884858 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:09.884830    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:09.984853 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:09.984825    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:30.037225 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:30.037192    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:33.832953 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:33.832902    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:35.778070 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:35.778038    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"]
Sep 16 04:57:35.781350 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:35.781320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:57:35.784304 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:35.784275    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9sq7l\""
Sep 16 04:57:35.784889 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:35.784871    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 04:57:35.785010 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:35.784908    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 04:57:35.797414 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:35.797392    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"]
Sep 16 04:57:35.802475 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:35.802454    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fp4v\" (UniqueName: \"kubernetes.io/projected/e973ab4b-a664-4175-ba04-555069ba7bf8-kube-api-access-2fp4v\") pod \"limitador-operator-controller-manager-74495c69f8-42v49\" (UID: \"e973ab4b-a664-4175-ba04-555069ba7bf8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:57:35.903533 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:35.903502    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fp4v\" (UniqueName: \"kubernetes.io/projected/e973ab4b-a664-4175-ba04-555069ba7bf8-kube-api-access-2fp4v\") pod \"limitador-operator-controller-manager-74495c69f8-42v49\" (UID: \"e973ab4b-a664-4175-ba04-555069ba7bf8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:57:35.915784 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:35.915759    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fp4v\" (UniqueName: \"kubernetes.io/projected/e973ab4b-a664-4175-ba04-555069ba7bf8-kube-api-access-2fp4v\") pod \"limitador-operator-controller-manager-74495c69f8-42v49\" (UID: \"e973ab4b-a664-4175-ba04-555069ba7bf8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:57:36.091137 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:36.091048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:57:36.228907 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:36.228878    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:36.366457 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:36.366325    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"]
Sep 16 04:57:36.369232 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:57:36.369201    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode973ab4b_a664_4175_ba04_555069ba7bf8.slice/crio-114f78a22cf01c9117fc2bbedf2515554ab9afcf4e13124a7d1ec0c8c57ab220 WatchSource:0}: Error finding container 114f78a22cf01c9117fc2bbedf2515554ab9afcf4e13124a7d1ec0c8c57ab220: Status 404 returned error can't find the container with id 114f78a22cf01c9117fc2bbedf2515554ab9afcf4e13124a7d1ec0c8c57ab220
Sep 16 04:57:36.735383 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:36.735351    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49" event={"ID":"e973ab4b-a664-4175-ba04-555069ba7bf8","Type":"ContainerStarted","Data":"114f78a22cf01c9117fc2bbedf2515554ab9afcf4e13124a7d1ec0c8c57ab220"}
Sep 16 04:57:36.827120 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:36.827091    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:38.750036 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:38.749997    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49" event={"ID":"e973ab4b-a664-4175-ba04-555069ba7bf8","Type":"ContainerStarted","Data":"188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881"}
Sep 16 04:57:38.750472 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:38.750082    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:57:38.773572 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:38.773527    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49" podStartSLOduration=1.574923125 podStartE2EDuration="3.773511449s" totalImagesPullingTime="2.198588324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:57:38.772459418 +0000 UTC m=+522.438389005" watchObservedRunningTime="2026-09-16 04:57:38.773511449 +0000 UTC m=+522.439441041"
Sep 16 04:57:40.033649 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:40.033617    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:40.622803 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:40.622754    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:41.231430 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:41.231375    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:45.250304 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.250271    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"]
Sep 16 04:57:45.253592 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.253573    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:57:45.257943 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.257922    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l67nf\""
Sep 16 04:57:45.266751 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.266722    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/119a3e25-689f-4b04-b846-b346da6fd0f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" (UID: \"119a3e25-689f-4b04-b846-b346da6fd0f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:57:45.266856 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.266779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj7k9\" (UniqueName: \"kubernetes.io/projected/119a3e25-689f-4b04-b846-b346da6fd0f6-kube-api-access-gj7k9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" (UID: \"119a3e25-689f-4b04-b846-b346da6fd0f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:57:45.267171 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.267151    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"]
Sep 16 04:57:45.367178 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.367148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/119a3e25-689f-4b04-b846-b346da6fd0f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" (UID: \"119a3e25-689f-4b04-b846-b346da6fd0f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:57:45.367363 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.367199    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj7k9\" (UniqueName: \"kubernetes.io/projected/119a3e25-689f-4b04-b846-b346da6fd0f6-kube-api-access-gj7k9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" (UID: \"119a3e25-689f-4b04-b846-b346da6fd0f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:57:45.367565 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.367541    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/119a3e25-689f-4b04-b846-b346da6fd0f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" (UID: \"119a3e25-689f-4b04-b846-b346da6fd0f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:57:45.382930 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.382908    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj7k9\" (UniqueName: \"kubernetes.io/projected/119a3e25-689f-4b04-b846-b346da6fd0f6-kube-api-access-gj7k9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" (UID: \"119a3e25-689f-4b04-b846-b346da6fd0f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:57:45.563009 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.562920    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:57:45.844695 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:45.844671    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"]
Sep 16 04:57:45.847698 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:57:45.847668    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119a3e25_689f_4b04_b846_b346da6fd0f6.slice/crio-fc8b7d9e255001477044621fc6b190d6212184a84a2eabc352e48f901c80e107 WatchSource:0}: Error finding container fc8b7d9e255001477044621fc6b190d6212184a84a2eabc352e48f901c80e107: Status 404 returned error can't find the container with id fc8b7d9e255001477044621fc6b190d6212184a84a2eabc352e48f901c80e107
Sep 16 04:57:46.786448 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:46.786416    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc" event={"ID":"119a3e25-689f-4b04-b846-b346da6fd0f6","Type":"ContainerStarted","Data":"fc8b7d9e255001477044621fc6b190d6212184a84a2eabc352e48f901c80e107"}
Sep 16 04:57:49.755749 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:49.755714    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:57:49.805744 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:49.805713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc" event={"ID":"119a3e25-689f-4b04-b846-b346da6fd0f6","Type":"ContainerStarted","Data":"10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629"}
Sep 16 04:57:49.805918 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:49.805859    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:57:49.823973 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:49.823853    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc" podStartSLOduration=1.106026656 podStartE2EDuration="4.823833811s" totalImagesPullingTime="3.717807155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:57:49.82248783 +0000 UTC m=+533.488417443" watchObservedRunningTime="2026-09-16 04:57:49.823833811 +0000 UTC m=+533.489763405"
Sep 16 04:57:52.070817 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:52.070784    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:52.760386 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:52.760354    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:55.115100 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:55.115069    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:55.633040 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:55.633010    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:55.733697 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:55.733672    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:56.030867 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:56.030800    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:56.134198 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:56.134174    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:56.278289 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:56.278260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:57.462181 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:57.462136    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:58.493333 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:58.493288    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:57:58.629003 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:57:58.628959    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:00.811131 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:00.811096    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:01.522104 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:01.522078    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:01.656064 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:01.656033    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:01.859318 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:01.859263    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:02.011431 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.011412    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:02.082153 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.082130    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"]
Sep 16 04:58:02.123949 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.123896    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"]
Sep 16 04:58:02.124037 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.123975    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:02.175827 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.175806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zssq4\" (UniqueName: \"kubernetes.io/projected/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-kube-api-access-zssq4\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6nslm\" (UID: \"b2cc71c7-f21d-4a9a-8e3f-1afe84021219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:02.175957 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.175895    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6nslm\" (UID: \"b2cc71c7-f21d-4a9a-8e3f-1afe84021219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:02.193466 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.193420    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"]
Sep 16 04:58:02.193643 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:02.193626    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-zssq4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm" podUID="b2cc71c7-f21d-4a9a-8e3f-1afe84021219"
Sep 16 04:58:02.243227 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.243200    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"]
Sep 16 04:58:02.265504 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.265483    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"]
Sep 16 04:58:02.265578 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.265572    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:02.276810 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.276789    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6nslm\" (UID: \"b2cc71c7-f21d-4a9a-8e3f-1afe84021219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:02.276889 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.276825    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zssq4\" (UniqueName: \"kubernetes.io/projected/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-kube-api-access-zssq4\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6nslm\" (UID: \"b2cc71c7-f21d-4a9a-8e3f-1afe84021219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:02.277100 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.277082    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6nslm\" (UID: \"b2cc71c7-f21d-4a9a-8e3f-1afe84021219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:02.287118 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.287096    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:02.287232 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.287192    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zssq4\" (UniqueName: \"kubernetes.io/projected/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-kube-api-access-zssq4\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6nslm\" (UID: \"b2cc71c7-f21d-4a9a-8e3f-1afe84021219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:02.377795 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.377771    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" (UID: \"ad254d7a-1f93-4ce9-b1ec-8979b8068f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:02.377888 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.377805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2pcn\" (UniqueName: \"kubernetes.io/projected/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-kube-api-access-r2pcn\") pod \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" (UID: \"ad254d7a-1f93-4ce9-b1ec-8979b8068f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:02.381144 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.381124    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:02.478442 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.478416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" (UID: \"ad254d7a-1f93-4ce9-b1ec-8979b8068f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:02.478537 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.478446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2pcn\" (UniqueName: \"kubernetes.io/projected/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-kube-api-access-r2pcn\") pod \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" (UID: \"ad254d7a-1f93-4ce9-b1ec-8979b8068f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:02.478775 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.478758    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" (UID: \"ad254d7a-1f93-4ce9-b1ec-8979b8068f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:02.487915 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.487896    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2pcn\" (UniqueName: \"kubernetes.io/projected/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-kube-api-access-r2pcn\") pod \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" (UID: \"ad254d7a-1f93-4ce9-b1ec-8979b8068f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:02.575438 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.575414    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:02.859253 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.859222    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:02.878079 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.878059    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"]
Sep 16 04:58:02.880150 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.880127    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:02.881301 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:58:02.881272    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad254d7a_1f93_4ce9_b1ec_8979b8068f17.slice/crio-8da880da0e6e2c487a688ee5d7f2907ca958c9343b1c2953203b9fc1419cdb10 WatchSource:0}: Error finding container 8da880da0e6e2c487a688ee5d7f2907ca958c9343b1c2953203b9fc1419cdb10: Status 404 returned error can't find the container with id 8da880da0e6e2c487a688ee5d7f2907ca958c9343b1c2953203b9fc1419cdb10
Sep 16 04:58:02.981232 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.981213    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zssq4\" (UniqueName: \"kubernetes.io/projected/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-kube-api-access-zssq4\") pod \"b2cc71c7-f21d-4a9a-8e3f-1afe84021219\" (UID: \"b2cc71c7-f21d-4a9a-8e3f-1afe84021219\") "
Sep 16 04:58:02.981315 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.981249    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-extensions-socket-volume\") pod \"b2cc71c7-f21d-4a9a-8e3f-1afe84021219\" (UID: \"b2cc71c7-f21d-4a9a-8e3f-1afe84021219\") "
Sep 16 04:58:02.981508 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.981485    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b2cc71c7-f21d-4a9a-8e3f-1afe84021219" (UID: "b2cc71c7-f21d-4a9a-8e3f-1afe84021219"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 04:58:02.983114 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:02.983080    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-kube-api-access-zssq4" (OuterVolumeSpecName: "kube-api-access-zssq4") pod "b2cc71c7-f21d-4a9a-8e3f-1afe84021219" (UID: "b2cc71c7-f21d-4a9a-8e3f-1afe84021219"). InnerVolumeSpecName "kube-api-access-zssq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 04:58:03.050634 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.050609    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"]
Sep 16 04:58:03.055050 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.055028    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"]
Sep 16 04:58:03.063001 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.062980    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"]
Sep 16 04:58:03.072985 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.072965    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"]
Sep 16 04:58:03.078375 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.078356    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"]
Sep 16 04:58:03.078642 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.078628    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" containerName="manager"
Sep 16 04:58:03.078642 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.078642    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" containerName="manager"
Sep 16 04:58:03.078773 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.078690    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" containerName="manager"
Sep 16 04:58:03.082275 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.082256    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zssq4\" (UniqueName: \"kubernetes.io/projected/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-kube-api-access-zssq4\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:03.082275 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.082275    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2cc71c7-f21d-4a9a-8e3f-1afe84021219-extensions-socket-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:03.100304 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.100272    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"]
Sep 16 04:58:03.100407 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.100313    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"]
Sep 16 04:58:03.100407 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.100400    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:03.100518 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.100453    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" containerName="manager" containerID="cri-o://10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629" gracePeriod=2
Sep 16 04:58:03.106315 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.106288    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"]
Sep 16 04:58:03.125431 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.125409    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"]
Sep 16 04:58:03.125653 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.125626    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" containerName="manager" containerID="cri-o://188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881" gracePeriod=2
Sep 16 04:58:03.127669 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.127609    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"]
Sep 16 04:58:03.128148 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.128130    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" containerName="manager"
Sep 16 04:58:03.128216 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.128154    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" containerName="manager"
Sep 16 04:58:03.128259 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.128229    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" containerName="manager"
Sep 16 04:58:03.131544 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.131155    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.139623 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.133849    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-42v49\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:03.139623 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.135486    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.150523 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.150504    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"]
Sep 16 04:58:03.150633 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.150542    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"]
Sep 16 04:58:03.150680 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.150642    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:03.153092 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.153047    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.160884 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.160862    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k"]
Sep 16 04:58:03.161253 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.161231    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" containerName="manager"
Sep 16 04:58:03.161253 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.161254    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" containerName="manager"
Sep 16 04:58:03.161431 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.161344    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" containerName="manager"
Sep 16 04:58:03.178752 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.178728    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k"]
Sep 16 04:58:03.178865 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.178808    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k"
Sep 16 04:58:03.179585 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.179559    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-42v49\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:03.181123 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.181094    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-42v49\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:03.182877 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.182853    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.183037 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.183005    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-qbtf5\" (UID: \"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:03.183145 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.183071    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkrw\" (UniqueName: \"kubernetes.io/projected/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-kube-api-access-kqkrw\") pod \"kuadrant-operator-controller-manager-5f685c496f-qbtf5\" (UID: \"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:03.183202 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.183146    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3948b2b-b98e-435a-9b77-0f5ab7500f60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9lqw6\" (UID: \"b3948b2b-b98e-435a-9b77-0f5ab7500f60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:03.183259 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.183200    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksnp5\" (UniqueName: \"kubernetes.io/projected/b3948b2b-b98e-435a-9b77-0f5ab7500f60-kube-api-access-ksnp5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9lqw6\" (UID: \"b3948b2b-b98e-435a-9b77-0f5ab7500f60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:03.284539 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.284504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkrw\" (UniqueName: \"kubernetes.io/projected/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-kube-api-access-kqkrw\") pod \"kuadrant-operator-controller-manager-5f685c496f-qbtf5\" (UID: \"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:03.284707 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.284549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5xj\" (UniqueName: \"kubernetes.io/projected/9224eeff-9a2a-46e5-977b-2a0cdd96d9db-kube-api-access-ct5xj\") pod \"limitador-operator-controller-manager-74495c69f8-rds7k\" (UID: \"9224eeff-9a2a-46e5-977b-2a0cdd96d9db\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k"
Sep 16 04:58:03.284707 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.284606    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3948b2b-b98e-435a-9b77-0f5ab7500f60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9lqw6\" (UID: \"b3948b2b-b98e-435a-9b77-0f5ab7500f60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:03.284837 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.284694    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksnp5\" (UniqueName: \"kubernetes.io/projected/b3948b2b-b98e-435a-9b77-0f5ab7500f60-kube-api-access-ksnp5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9lqw6\" (UID: \"b3948b2b-b98e-435a-9b77-0f5ab7500f60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:03.284837 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.284764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-qbtf5\" (UID: \"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:03.285051 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.285022    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3948b2b-b98e-435a-9b77-0f5ab7500f60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9lqw6\" (UID: \"b3948b2b-b98e-435a-9b77-0f5ab7500f60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:03.285200 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.285178    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-qbtf5\" (UID: \"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:03.294582 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.294561    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksnp5\" (UniqueName: \"kubernetes.io/projected/b3948b2b-b98e-435a-9b77-0f5ab7500f60-kube-api-access-ksnp5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9lqw6\" (UID: \"b3948b2b-b98e-435a-9b77-0f5ab7500f60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:03.295017 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.294992    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkrw\" (UniqueName: \"kubernetes.io/projected/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-kube-api-access-kqkrw\") pod \"kuadrant-operator-controller-manager-5f685c496f-qbtf5\" (UID: \"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:03.385414 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.385393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5xj\" (UniqueName: \"kubernetes.io/projected/9224eeff-9a2a-46e5-977b-2a0cdd96d9db-kube-api-access-ct5xj\") pod \"limitador-operator-controller-manager-74495c69f8-rds7k\" (UID: \"9224eeff-9a2a-46e5-977b-2a0cdd96d9db\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k"
Sep 16 04:58:03.402654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.402634    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.405107 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.405063    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-42v49\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:03.408831 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.408811    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.412958 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.412941    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:03.416657 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.416644    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:03.418851 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.418834    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.421523 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.421505    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-42v49\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:03.425724 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.425707    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5xj\" (UniqueName: \"kubernetes.io/projected/9224eeff-9a2a-46e5-977b-2a0cdd96d9db-kube-api-access-ct5xj\") pod \"limitador-operator-controller-manager-74495c69f8-rds7k\" (UID: \"9224eeff-9a2a-46e5-977b-2a0cdd96d9db\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k"
Sep 16 04:58:03.485976 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.485955    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/119a3e25-689f-4b04-b846-b346da6fd0f6-extensions-socket-volume\") pod \"119a3e25-689f-4b04-b846-b346da6fd0f6\" (UID: \"119a3e25-689f-4b04-b846-b346da6fd0f6\") "
Sep 16 04:58:03.486081 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.486014    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fp4v\" (UniqueName: \"kubernetes.io/projected/e973ab4b-a664-4175-ba04-555069ba7bf8-kube-api-access-2fp4v\") pod \"e973ab4b-a664-4175-ba04-555069ba7bf8\" (UID: \"e973ab4b-a664-4175-ba04-555069ba7bf8\") "
Sep 16 04:58:03.486081 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.486064    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj7k9\" (UniqueName: \"kubernetes.io/projected/119a3e25-689f-4b04-b846-b346da6fd0f6-kube-api-access-gj7k9\") pod \"119a3e25-689f-4b04-b846-b346da6fd0f6\" (UID: \"119a3e25-689f-4b04-b846-b346da6fd0f6\") "
Sep 16 04:58:03.486332 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.486307    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119a3e25-689f-4b04-b846-b346da6fd0f6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "119a3e25-689f-4b04-b846-b346da6fd0f6" (UID: "119a3e25-689f-4b04-b846-b346da6fd0f6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 04:58:03.488798 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.488773    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119a3e25-689f-4b04-b846-b346da6fd0f6-kube-api-access-gj7k9" (OuterVolumeSpecName: "kube-api-access-gj7k9") pod "119a3e25-689f-4b04-b846-b346da6fd0f6" (UID: "119a3e25-689f-4b04-b846-b346da6fd0f6"). InnerVolumeSpecName "kube-api-access-gj7k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 04:58:03.488884 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.488817    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e973ab4b-a664-4175-ba04-555069ba7bf8-kube-api-access-2fp4v" (OuterVolumeSpecName: "kube-api-access-2fp4v") pod "e973ab4b-a664-4175-ba04-555069ba7bf8" (UID: "e973ab4b-a664-4175-ba04-555069ba7bf8"). InnerVolumeSpecName "kube-api-access-2fp4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 04:58:03.545957 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.545935    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:03.552037 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.552021    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k"
Sep 16 04:58:03.586936 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.586913    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj7k9\" (UniqueName: \"kubernetes.io/projected/119a3e25-689f-4b04-b846-b346da6fd0f6-kube-api-access-gj7k9\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:03.586936 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.586934    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/119a3e25-689f-4b04-b846-b346da6fd0f6-extensions-socket-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:03.587086 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.586943    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fp4v\" (UniqueName: \"kubernetes.io/projected/e973ab4b-a664-4175-ba04-555069ba7bf8-kube-api-access-2fp4v\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:03.705193 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.705131    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"]
Sep 16 04:58:03.709823 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:58:03.709801    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5250c1_c51a_45f8_8740_83bb6b3d6aa2.slice/crio-8c147ecc15453d416192f5f42a3dc08090fca9337fef7fc8104a98c633d0ec91 WatchSource:0}: Error finding container 8c147ecc15453d416192f5f42a3dc08090fca9337fef7fc8104a98c633d0ec91: Status 404 returned error can't find the container with id 8c147ecc15453d416192f5f42a3dc08090fca9337fef7fc8104a98c633d0ec91
Sep 16 04:58:03.844691 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.844671    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"]
Sep 16 04:58:03.846931 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:58:03.846903    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3948b2b_b98e_435a_9b77_0f5ab7500f60.slice/crio-fd18ab25c9d8312a61d95a02dcbf34880cb30db09b3e4099c98ccfea31b084ec WatchSource:0}: Error finding container fd18ab25c9d8312a61d95a02dcbf34880cb30db09b3e4099c98ccfea31b084ec: Status 404 returned error can't find the container with id fd18ab25c9d8312a61d95a02dcbf34880cb30db09b3e4099c98ccfea31b084ec
Sep 16 04:58:03.864802 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.864772    2573 generic.go:356] "Generic (PLEG): container finished" podID="e973ab4b-a664-4175-ba04-555069ba7bf8" containerID="188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881" exitCode=0
Sep 16 04:58:03.864901 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.864812    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:03.864901 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.864832    2573 scope.go:117] "RemoveContainer" containerID="188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881"
Sep 16 04:58:03.866410 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.866386    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6" event={"ID":"b3948b2b-b98e-435a-9b77-0f5ab7500f60","Type":"ContainerStarted","Data":"fd18ab25c9d8312a61d95a02dcbf34880cb30db09b3e4099c98ccfea31b084ec"}
Sep 16 04:58:03.867883 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.867858    2573 generic.go:356] "Generic (PLEG): container finished" podID="119a3e25-689f-4b04-b846-b346da6fd0f6" containerID="10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629" exitCode=0
Sep 16 04:58:03.867970 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.867896    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.868135 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.868110    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-42v49\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:03.869779 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.869750    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.870529 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.870512    2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r" container="manager"
Sep 16 04:58:03.871304 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.871275    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-42v49\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:03.872348 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.872328    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:03.872724 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.872691    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5" event={"ID":"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2","Type":"ContainerStarted","Data":"f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0"}
Sep 16 04:58:03.872799 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.872741    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5" event={"ID":"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2","Type":"ContainerStarted","Data":"8c147ecc15453d416192f5f42a3dc08090fca9337fef7fc8104a98c633d0ec91"}
Sep 16 04:58:03.872799 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.872748    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.872799 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.872769    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:03.874317 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.874241    2573 scope.go:117] "RemoveContainer" containerID="188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881"
Sep 16 04:58:03.874643 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.874622    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881\": container with ID starting with 188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881 not found: ID does not exist" containerID="188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881"
Sep 16 04:58:03.874741 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.874654    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881"} err="failed to get container status \"188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881\": rpc error: code = NotFound desc = could not find container \"188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881\": container with ID starting with 188421c583b8d7d7e37a78425740118ac09343d5777eee877ac52e7c9637c881 not found: ID does not exist"
Sep 16 04:58:03.874741 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.874676    2573 scope.go:117] "RemoveContainer" containerID="10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629"
Sep 16 04:58:03.877556 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.877537    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k"]
Sep 16 04:58:03.881160 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:58:03.881105    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9224eeff_9a2a_46e5_977b_2a0cdd96d9db.slice/crio-5d9693f62bf43b0c87262a5bc57ec6d349e429d5ec01c5d3a99f5d1c383d8415 WatchSource:0}: Error finding container 5d9693f62bf43b0c87262a5bc57ec6d349e429d5ec01c5d3a99f5d1c383d8415: Status 404 returned error can't find the container with id 5d9693f62bf43b0c87262a5bc57ec6d349e429d5ec01c5d3a99f5d1c383d8415
Sep 16 04:58:03.885690 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.885668    2573 scope.go:117] "RemoveContainer" containerID="10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629"
Sep 16 04:58:03.885981 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.885951    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629\": container with ID starting with 10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629 not found: ID does not exist" containerID="10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629"
Sep 16 04:58:03.886027 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.885983    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629"} err="failed to get container status \"10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629\": rpc error: code = NotFound desc = could not find container \"10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629\": container with ID starting with 10c6327f4e69cc0c37da1c5c6e10a7000b365957439237e8e753f59021c90629 not found: ID does not exist"
Sep 16 04:58:03.891818 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:03.891782    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5" podStartSLOduration=0.891769684 podStartE2EDuration="891.769684ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:58:03.890895618 +0000 UTC m=+547.556825232" watchObservedRunningTime="2026-09-16 04:58:03.891769684 +0000 UTC m=+547.557699277"
Sep 16 04:58:03.893275 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.893252    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-6nslm\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="b2cc71c7-f21d-4a9a-8e3f-1afe84021219" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:03.894847 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.894822    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:03.896197 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.896171    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-42v49\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:03.897913 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.897892    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.899311 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.899285    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-x29hc\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-x29hc"
Sep 16 04:58:03.901045 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.900863    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-6nslm\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="b2cc71c7-f21d-4a9a-8e3f-1afe84021219" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6nslm"
Sep 16 04:58:03.902240 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.902220    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:03.903609 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:03.903575    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-42v49\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-42v49"
Sep 16 04:58:04.786345 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:04.786318    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="119a3e25-689f-4b04-b846-b346da6fd0f6" path="/var/lib/kubelet/pods/119a3e25-689f-4b04-b846-b346da6fd0f6/volumes"
Sep 16 04:58:04.786688 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:04.786673    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2cc71c7-f21d-4a9a-8e3f-1afe84021219" path="/var/lib/kubelet/pods/b2cc71c7-f21d-4a9a-8e3f-1afe84021219/volumes"
Sep 16 04:58:04.786884 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:04.786873    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e973ab4b-a664-4175-ba04-555069ba7bf8" path="/var/lib/kubelet/pods/e973ab4b-a664-4175-ba04-555069ba7bf8/volumes"
Sep 16 04:58:04.879540 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:04.879498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k" event={"ID":"9224eeff-9a2a-46e5-977b-2a0cdd96d9db","Type":"ContainerStarted","Data":"4fde7e3cec4e37b291b913f679089986ea3d12a84c337ee95888066f6d4bf796"}
Sep 16 04:58:04.879732 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:04.879561    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k"
Sep 16 04:58:04.879732 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:04.879588    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k" event={"ID":"9224eeff-9a2a-46e5-977b-2a0cdd96d9db","Type":"ContainerStarted","Data":"5d9693f62bf43b0c87262a5bc57ec6d349e429d5ec01c5d3a99f5d1c383d8415"}
Sep 16 04:58:04.881259 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:04.881233    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" containerName="manager" containerID="cri-o://48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3" gracePeriod=2
Sep 16 04:58:04.883892 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:04.883865    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6" event={"ID":"b3948b2b-b98e-435a-9b77-0f5ab7500f60","Type":"ContainerStarted","Data":"670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e"}
Sep 16 04:58:04.923821 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:04.923771    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k" podStartSLOduration=1.9237524160000001 podStartE2EDuration="1.923752416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:58:04.902816746 +0000 UTC m=+548.568746338" watchObservedRunningTime="2026-09-16 04:58:04.923752416 +0000 UTC m=+548.589682010"
Sep 16 04:58:04.924670 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:04.924635    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6" podStartSLOduration=1.924625438 podStartE2EDuration="1.924625438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:58:04.922849044 +0000 UTC m=+548.588778657" watchObservedRunningTime="2026-09-16 04:58:04.924625438 +0000 UTC m=+548.590555031"
Sep 16 04:58:05.113136 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.113109    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:05.115189 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:05.115161    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:05.197621 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.197581    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2pcn\" (UniqueName: \"kubernetes.io/projected/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-kube-api-access-r2pcn\") pod \"ad254d7a-1f93-4ce9-b1ec-8979b8068f17\" (UID: \"ad254d7a-1f93-4ce9-b1ec-8979b8068f17\") "
Sep 16 04:58:05.197731 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.197691    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-extensions-socket-volume\") pod \"ad254d7a-1f93-4ce9-b1ec-8979b8068f17\" (UID: \"ad254d7a-1f93-4ce9-b1ec-8979b8068f17\") "
Sep 16 04:58:05.197930 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.197911    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ad254d7a-1f93-4ce9-b1ec-8979b8068f17" (UID: "ad254d7a-1f93-4ce9-b1ec-8979b8068f17"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 04:58:05.199550 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.199528    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-kube-api-access-r2pcn" (OuterVolumeSpecName: "kube-api-access-r2pcn") pod "ad254d7a-1f93-4ce9-b1ec-8979b8068f17" (UID: "ad254d7a-1f93-4ce9-b1ec-8979b8068f17"). InnerVolumeSpecName "kube-api-access-r2pcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 04:58:05.298164 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.298138    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-extensions-socket-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:05.298164 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.298161    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2pcn\" (UniqueName: \"kubernetes.io/projected/ad254d7a-1f93-4ce9-b1ec-8979b8068f17-kube-api-access-r2pcn\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:05.896011 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.895977    2573 generic.go:356] "Generic (PLEG): container finished" podID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" containerID="48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3" exitCode=2
Sep 16 04:58:05.896425 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.896042    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:05.896425 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.896075    2573 scope.go:117] "RemoveContainer" containerID="48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3"
Sep 16 04:58:05.896547 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.896437    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:05.897926 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:05.897894    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:05.906411 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:05.906385    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:05.906490 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.906411    2573 scope.go:117] "RemoveContainer" containerID="48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3"
Sep 16 04:58:05.906742 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:05.906722    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3\": container with ID starting with 48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3 not found: ID does not exist" containerID="48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3"
Sep 16 04:58:05.906817 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:05.906749    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3"} err="failed to get container status \"48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3\": rpc error: code = NotFound desc = could not find container \"48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3\": container with ID starting with 48704929ca22c0aba163ec321a0e760dbc56dafbe100bb9da590e47023ee18b3 not found: ID does not exist"
Sep 16 04:58:06.783814 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:06.783777    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-2kg6r\" is forbidden: User \"system:node:ip-10-0-133-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-117.ec2.internal' and this object" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-2kg6r"
Sep 16 04:58:06.786654 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:06.786631    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad254d7a-1f93-4ce9-b1ec-8979b8068f17" path="/var/lib/kubelet/pods/ad254d7a-1f93-4ce9-b1ec-8979b8068f17/volumes"
Sep 16 04:58:07.564937 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:07.564904    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:07.633384 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:07.633359    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:08.028328 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:08.028298    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:08.133345 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:08.133326    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:10.888242 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:10.888212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:10.987907 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:10.987883    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:14.886543 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:14.886512    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:14.946278 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:14.946245    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"]
Sep 16 04:58:14.946514 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:14.946465    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6" podUID="b3948b2b-b98e-435a-9b77-0f5ab7500f60" containerName="manager" containerID="cri-o://670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e" gracePeriod=10
Sep 16 04:58:14.948315 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:14.948291    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:15.188816 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.188794    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:15.268461 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.268440    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksnp5\" (UniqueName: \"kubernetes.io/projected/b3948b2b-b98e-435a-9b77-0f5ab7500f60-kube-api-access-ksnp5\") pod \"b3948b2b-b98e-435a-9b77-0f5ab7500f60\" (UID: \"b3948b2b-b98e-435a-9b77-0f5ab7500f60\") "
Sep 16 04:58:15.268582 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.268508    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3948b2b-b98e-435a-9b77-0f5ab7500f60-extensions-socket-volume\") pod \"b3948b2b-b98e-435a-9b77-0f5ab7500f60\" (UID: \"b3948b2b-b98e-435a-9b77-0f5ab7500f60\") "
Sep 16 04:58:15.268889 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.268865    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3948b2b-b98e-435a-9b77-0f5ab7500f60-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b3948b2b-b98e-435a-9b77-0f5ab7500f60" (UID: "b3948b2b-b98e-435a-9b77-0f5ab7500f60"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 04:58:15.270235 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.270213    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3948b2b-b98e-435a-9b77-0f5ab7500f60-kube-api-access-ksnp5" (OuterVolumeSpecName: "kube-api-access-ksnp5") pod "b3948b2b-b98e-435a-9b77-0f5ab7500f60" (UID: "b3948b2b-b98e-435a-9b77-0f5ab7500f60"). InnerVolumeSpecName "kube-api-access-ksnp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 04:58:15.340750 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.340724    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"]
Sep 16 04:58:15.341015 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.341003    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3948b2b-b98e-435a-9b77-0f5ab7500f60" containerName="manager"
Sep 16 04:58:15.341015 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.341016    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3948b2b-b98e-435a-9b77-0f5ab7500f60" containerName="manager"
Sep 16 04:58:15.341114 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.341104    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3948b2b-b98e-435a-9b77-0f5ab7500f60" containerName="manager"
Sep 16 04:58:15.343013 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.342996    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 04:58:15.362317 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.362296    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"]
Sep 16 04:58:15.369502 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.369483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fvkwg\" (UID: \"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 04:58:15.369610 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.369520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cntz\" (UniqueName: \"kubernetes.io/projected/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-kube-api-access-6cntz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fvkwg\" (UID: \"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 04:58:15.369678 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.369631    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3948b2b-b98e-435a-9b77-0f5ab7500f60-extensions-socket-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:15.369678 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.369648    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksnp5\" (UniqueName: \"kubernetes.io/projected/b3948b2b-b98e-435a-9b77-0f5ab7500f60-kube-api-access-ksnp5\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:15.470870 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.470824    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fvkwg\" (UID: \"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 04:58:15.470870 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.470851    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cntz\" (UniqueName: \"kubernetes.io/projected/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-kube-api-access-6cntz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fvkwg\" (UID: \"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 04:58:15.471175 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.471159    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fvkwg\" (UID: \"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 04:58:15.481845 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.481821    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cntz\" (UniqueName: \"kubernetes.io/projected/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-kube-api-access-6cntz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fvkwg\" (UID: \"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 04:58:15.652585 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.652562    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 04:58:15.689107 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.689085    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:15.788371 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.788312    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:15.899003 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.898980    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rds7k"
Sep 16 04:58:15.931025 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.930999    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"]
Sep 16 04:58:15.934768 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:58:15.934745    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6b0b0f_ecfc_4441_8ac3_f37f11cf3003.slice/crio-c8e5c6f2363f87b135f2b16c824d5dc2a9bae1393498528cf8145ad094d032a7 WatchSource:0}: Error finding container c8e5c6f2363f87b135f2b16c824d5dc2a9bae1393498528cf8145ad094d032a7: Status 404 returned error can't find the container with id c8e5c6f2363f87b135f2b16c824d5dc2a9bae1393498528cf8145ad094d032a7
Sep 16 04:58:15.953322 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.953289    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3948b2b-b98e-435a-9b77-0f5ab7500f60" containerID="670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e" exitCode=0
Sep 16 04:58:15.953410 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.953381    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6" event={"ID":"b3948b2b-b98e-435a-9b77-0f5ab7500f60","Type":"ContainerDied","Data":"670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e"}
Sep 16 04:58:15.953482 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.953445    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"
Sep 16 04:58:15.953482 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.953458    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6" event={"ID":"b3948b2b-b98e-435a-9b77-0f5ab7500f60","Type":"ContainerDied","Data":"fd18ab25c9d8312a61d95a02dcbf34880cb30db09b3e4099c98ccfea31b084ec"}
Sep 16 04:58:15.953575 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.953492    2573 scope.go:117] "RemoveContainer" containerID="670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e"
Sep 16 04:58:15.961856 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.955554    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg" event={"ID":"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003","Type":"ContainerStarted","Data":"c8e5c6f2363f87b135f2b16c824d5dc2a9bae1393498528cf8145ad094d032a7"}
Sep 16 04:58:15.975639 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.975624    2573 scope.go:117] "RemoveContainer" containerID="670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e"
Sep 16 04:58:15.975912 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:15.975893    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e\": container with ID starting with 670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e not found: ID does not exist" containerID="670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e"
Sep 16 04:58:15.975983 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:15.975923    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e"} err="failed to get container status \"670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e\": rpc error: code = NotFound desc = could not find container \"670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e\": container with ID starting with 670a827fb387d2cc0126f179d883e8b88b3d101cfdd32824673472b57552010e not found: ID does not exist"
Sep 16 04:58:16.004511 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:16.004490    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"]
Sep 16 04:58:16.015855 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:16.015832    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9lqw6"]
Sep 16 04:58:16.785528 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:16.785500    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3948b2b-b98e-435a-9b77-0f5ab7500f60" path="/var/lib/kubelet/pods/b3948b2b-b98e-435a-9b77-0f5ab7500f60/volumes"
Sep 16 04:58:16.962408 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:16.962383    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg" event={"ID":"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003","Type":"ContainerStarted","Data":"8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f"}
Sep 16 04:58:16.962771 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:16.962512    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 04:58:16.987292 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:16.987250    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg" podStartSLOduration=1.98723705 podStartE2EDuration="1.98723705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:58:16.985718313 +0000 UTC m=+560.651647915" watchObservedRunningTime="2026-09-16 04:58:16.98723705 +0000 UTC m=+560.653166646"
Sep 16 04:58:20.437350 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:20.437312    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:20.578197 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:20.578169    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:20.671404 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:20.671371    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:27.059471 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:27.059440    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:27.157221 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:27.157193    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:27.968674 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:27.968645    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 04:58:28.033327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:28.028480    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"]
Sep 16 04:58:28.033327 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:28.029379    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5" podUID="7f5250c1-c51a-45f8-8740-83bb6b3d6aa2" containerName="manager" containerID="cri-o://f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0" gracePeriod=10
Sep 16 04:58:28.272254 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:28.272228    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:28.366089 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:28.366055    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqkrw\" (UniqueName: \"kubernetes.io/projected/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-kube-api-access-kqkrw\") pod \"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2\" (UID: \"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2\") "
Sep 16 04:58:28.366089 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:28.366093    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-extensions-socket-volume\") pod \"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2\" (UID: \"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2\") "
Sep 16 04:58:28.366506 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:28.366482    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7f5250c1-c51a-45f8-8740-83bb6b3d6aa2" (UID: "7f5250c1-c51a-45f8-8740-83bb6b3d6aa2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 04:58:28.368104 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:28.368082    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-kube-api-access-kqkrw" (OuterVolumeSpecName: "kube-api-access-kqkrw") pod "7f5250c1-c51a-45f8-8740-83bb6b3d6aa2" (UID: "7f5250c1-c51a-45f8-8740-83bb6b3d6aa2"). InnerVolumeSpecName "kube-api-access-kqkrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 04:58:28.467485 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:28.467450    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqkrw\" (UniqueName: \"kubernetes.io/projected/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-kube-api-access-kqkrw\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:28.467485 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:28.467479    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2-extensions-socket-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 04:58:29.017250 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:29.017214    2573 generic.go:356] "Generic (PLEG): container finished" podID="7f5250c1-c51a-45f8-8740-83bb6b3d6aa2" containerID="f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0" exitCode=0
Sep 16 04:58:29.017436 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:29.017261    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"
Sep 16 04:58:29.017436 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:29.017296    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5" event={"ID":"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2","Type":"ContainerDied","Data":"f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0"}
Sep 16 04:58:29.017436 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:29.017345    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5" event={"ID":"7f5250c1-c51a-45f8-8740-83bb6b3d6aa2","Type":"ContainerDied","Data":"8c147ecc15453d416192f5f42a3dc08090fca9337fef7fc8104a98c633d0ec91"}
Sep 16 04:58:29.017436 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:29.017373    2573 scope.go:117] "RemoveContainer" containerID="f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0"
Sep 16 04:58:29.025650 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:29.025632    2573 scope.go:117] "RemoveContainer" containerID="f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0"
Sep 16 04:58:29.025938 ip-10-0-133-117 kubenswrapper[2573]: E0916 04:58:29.025914    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0\": container with ID starting with f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0 not found: ID does not exist" containerID="f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0"
Sep 16 04:58:29.026012 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:29.025947    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0"} err="failed to get container status \"f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0\": rpc error: code = NotFound desc = could not find container \"f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0\": container with ID starting with f5e90acba1ebc157e1ecafb84493ffc6518f7f657141eb4892c82e689e24e3d0 not found: ID does not exist"
Sep 16 04:58:29.039574 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:29.039552    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"]
Sep 16 04:58:29.052046 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:29.051994    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-qbtf5"]
Sep 16 04:58:30.247677 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:30.247645    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:30.347732 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:30.347703    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:30.785190 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:30.785154    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f5250c1-c51a-45f8-8740-83bb6b3d6aa2" path="/var/lib/kubelet/pods/7f5250c1-c51a-45f8-8740-83bb6b3d6aa2/volumes"
Sep 16 04:58:31.389445 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:31.389412    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:31.587244 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:31.587213    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:32.832890 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:32.832863    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:34.702709 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:34.702667    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:34.805779 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:34.805732    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:34.903378 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:34.903349    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:39.853042 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:39.852965    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:39.952255 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:39.952228    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:40.053580 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:40.053541    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:48.176607 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.176575    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:48.283980 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.283947    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:48.381355 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.381323    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:48.499130 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.499069    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 04:58:48.499347 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.499336    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f5250c1-c51a-45f8-8740-83bb6b3d6aa2" containerName="manager"
Sep 16 04:58:48.499388 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.499349    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f5250c1-c51a-45f8-8740-83bb6b3d6aa2" containerName="manager"
Sep 16 04:58:48.499422 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.499399    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f5250c1-c51a-45f8-8740-83bb6b3d6aa2" containerName="manager"
Sep 16 04:58:48.503570 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.503551    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz"
Sep 16 04:58:48.505587 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.505558    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 04:58:48.505707 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.505653    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d2fln\""
Sep 16 04:58:48.510144 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.510126    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 04:58:48.534709 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.534687    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 04:58:48.608637 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.608592    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/917904f2-ff0d-49fe-bcf4-774129f33a89-config-file\") pod \"limitador-limitador-84994f7ffc-prqbz\" (UID: \"917904f2-ff0d-49fe-bcf4-774129f33a89\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz"
Sep 16 04:58:48.608728 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.608681    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpmt\" (UniqueName: \"kubernetes.io/projected/917904f2-ff0d-49fe-bcf4-774129f33a89-kube-api-access-wvpmt\") pod \"limitador-limitador-84994f7ffc-prqbz\" (UID: \"917904f2-ff0d-49fe-bcf4-774129f33a89\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz"
Sep 16 04:58:48.709698 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.709674    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/917904f2-ff0d-49fe-bcf4-774129f33a89-config-file\") pod \"limitador-limitador-84994f7ffc-prqbz\" (UID: \"917904f2-ff0d-49fe-bcf4-774129f33a89\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz"
Sep 16 04:58:48.709803 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.709714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpmt\" (UniqueName: \"kubernetes.io/projected/917904f2-ff0d-49fe-bcf4-774129f33a89-kube-api-access-wvpmt\") pod \"limitador-limitador-84994f7ffc-prqbz\" (UID: \"917904f2-ff0d-49fe-bcf4-774129f33a89\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz"
Sep 16 04:58:48.710235 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.710215    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/917904f2-ff0d-49fe-bcf4-774129f33a89-config-file\") pod \"limitador-limitador-84994f7ffc-prqbz\" (UID: \"917904f2-ff0d-49fe-bcf4-774129f33a89\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz"
Sep 16 04:58:48.718461 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.718436    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpmt\" (UniqueName: \"kubernetes.io/projected/917904f2-ff0d-49fe-bcf4-774129f33a89-kube-api-access-wvpmt\") pod \"limitador-limitador-84994f7ffc-prqbz\" (UID: \"917904f2-ff0d-49fe-bcf4-774129f33a89\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz"
Sep 16 04:58:48.814576 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:48.814529    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz"
Sep 16 04:58:49.015054 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:49.015027    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:49.086492 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:49.086433    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 04:58:49.089163 ip-10-0-133-117 kubenswrapper[2573]: W0916 04:58:49.089134    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917904f2_ff0d_49fe_bcf4_774129f33a89.slice/crio-52c070bb56275c586cec816beda382e4ebc4454ff85e53c38cbd719de2d8e641 WatchSource:0}: Error finding container 52c070bb56275c586cec816beda382e4ebc4454ff85e53c38cbd719de2d8e641: Status 404 returned error can't find the container with id 52c070bb56275c586cec816beda382e4ebc4454ff85e53c38cbd719de2d8e641
Sep 16 04:58:49.089520 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:49.089499    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:58:49.114635 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:49.114612    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz" event={"ID":"917904f2-ff0d-49fe-bcf4-774129f33a89","Type":"ContainerStarted","Data":"52c070bb56275c586cec816beda382e4ebc4454ff85e53c38cbd719de2d8e641"}
Sep 16 04:58:50.120266 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:50.120231    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz" event={"ID":"917904f2-ff0d-49fe-bcf4-774129f33a89","Type":"ContainerStarted","Data":"450a4c34fef467036a67a7c0909531b7cf764f63a825e996fe81a548faf2eb74"}
Sep 16 04:58:50.120586 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:50.120313    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz"
Sep 16 04:58:50.136186 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:50.136134    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz" podStartSLOduration=1.33662549 podStartE2EDuration="2.136118704s" totalImagesPullingTime="799.493214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:58:50.135056138 +0000 UTC m=+593.800985752" watchObservedRunningTime="2026-09-16 04:58:50.136118704 +0000 UTC m=+593.802048297"
Sep 16 04:58:56.717247 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:56.717214    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 04:58:56.720383 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:58:56.720364    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 04:59:01.126475 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:01.126439    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-prqbz"
Sep 16 04:59:04.942346 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:04.942310    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:05.030191 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:05.030154    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:05.452295 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:05.452260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:17.923133 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:17.923091    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:19.087583 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:19.087545    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:25.064280 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:25.064202    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:25.148381 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:25.148337    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:25.208453 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:25.208423    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:25.280902 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:25.280874    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:28.182187 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:28.182134    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:28.352119 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:28.352078    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:52.630884 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:52.630842    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 04:59:59.951056 ip-10-0-133-117 kubenswrapper[2573]: I0916 04:59:59.951013    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:00.049648 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:00.049616    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:01.682586 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:01.682551    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:01.855175 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:01.855147    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:01.955007 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:01.954948    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:02.115297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:02.115264    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:02.232196 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:02.232121    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:02.327156 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:02.327110    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:00:13.635583 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:13.635546    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:13.778614 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:13.778573    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:29.954470 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:29.954430    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:30.057448 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:30.057424    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:30.158670 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:30.158646    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:30.375184 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:30.375109    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:00:39.743185 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.743132    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"]
Sep 16 05:00:39.747088 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.747060    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.749746 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.749721    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 05:00:39.750886 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.750838    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 05:00:39.750886 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.750876    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sl6zs\""
Sep 16 05:00:39.751083 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.750841    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 16 05:00:39.755097 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.755074    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"]
Sep 16 05:00:39.787307 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.787233    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:39.865443 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.865416    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.865545 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.865464    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.865634 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.865550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdq5x\" (UniqueName: \"kubernetes.io/projected/38175809-aafd-43ee-9db6-719d5f516f7c-kube-api-access-tdq5x\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.865634 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.865619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.865746 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.865650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.865746 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.865675    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38175809-aafd-43ee-9db6-719d5f516f7c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.966772 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.966738    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.966882 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.966785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.966882 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.966819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdq5x\" (UniqueName: \"kubernetes.io/projected/38175809-aafd-43ee-9db6-719d5f516f7c-kube-api-access-tdq5x\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.966882 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.966856    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.967014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.966881    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.967014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.966903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38175809-aafd-43ee-9db6-719d5f516f7c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.967231 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.967210    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.967308 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.967275    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.967411 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.967329    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.969046 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.969031    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38175809-aafd-43ee-9db6-719d5f516f7c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.969161 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.969145    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38175809-aafd-43ee-9db6-719d5f516f7c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:39.974467 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:39.974442    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdq5x\" (UniqueName: \"kubernetes.io/projected/38175809-aafd-43ee-9db6-719d5f516f7c-kube-api-access-tdq5x\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vr5dh\" (UID: \"38175809-aafd-43ee-9db6-719d5f516f7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:40.060303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:40.060243    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:40.332919 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:40.332838    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"]
Sep 16 05:00:40.335642 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:00:40.335613    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38175809_aafd_43ee_9db6_719d5f516f7c.slice/crio-73149e76975b0ce0f3dc941ecdd95ca211cc01efc4d2d3c8783348fc9257a0ca WatchSource:0}: Error finding container 73149e76975b0ce0f3dc941ecdd95ca211cc01efc4d2d3c8783348fc9257a0ca: Status 404 returned error can't find the container with id 73149e76975b0ce0f3dc941ecdd95ca211cc01efc4d2d3c8783348fc9257a0ca
Sep 16 05:00:40.337009 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:40.336991    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 05:00:40.424726 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:40.424700    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:00:40.617287 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:40.617218    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh" event={"ID":"38175809-aafd-43ee-9db6-719d5f516f7c","Type":"ContainerStarted","Data":"73149e76975b0ce0f3dc941ecdd95ca211cc01efc4d2d3c8783348fc9257a0ca"}
Sep 16 05:00:41.628631 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:41.628574    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh" event={"ID":"38175809-aafd-43ee-9db6-719d5f516f7c","Type":"ContainerStarted","Data":"c20015daf31b4bb32ed95b8aa7443fb9d2cf2d3448176d18b6a42ac749648dfa"}
Sep 16 05:00:41.645073 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:41.645025    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh" podStartSLOduration=1.6947787010000002 podStartE2EDuration="2.645011758s" totalImagesPullingTime="950.233057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:00:41.644328636 +0000 UTC m=+705.310258251" watchObservedRunningTime="2026-09-16 05:00:41.645011758 +0000 UTC m=+705.310941342"
Sep 16 05:00:46.890695 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:46.890658    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:47.779419 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:47.779383    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:00:50.061183 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:50.061099    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:50.061183 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:50.061154    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:50.073474 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:50.073448    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:50.678872 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:50.678844    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vr5dh"
Sep 16 05:00:55.193746 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:55.193709    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:00:56.375638 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:00:56.375609    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:01:05.046659 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.046621    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"]
Sep 16 05:01:05.050034 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.049998    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.052428 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.052391    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 05:01:05.059657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.059633    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"]
Sep 16 05:01:05.075616 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.075576    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:01:05.142615 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.142571    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgtzn\" (UniqueName: \"kubernetes.io/projected/471b1d22-1296-4f99-aff8-a9a1c0e4604e-kube-api-access-qgtzn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.142726 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.142636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.142726 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.142665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.142726 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.142697    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.142861 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.142735    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.142861 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.142760    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471b1d22-1296-4f99-aff8-a9a1c0e4604e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.243911 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.243883    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgtzn\" (UniqueName: \"kubernetes.io/projected/471b1d22-1296-4f99-aff8-a9a1c0e4604e-kube-api-access-qgtzn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.244014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.243914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.244014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.243935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.244014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.243953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.244014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.243970    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.244014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.243992    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471b1d22-1296-4f99-aff8-a9a1c0e4604e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.244283 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.244258    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.244392 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.244373    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.244473 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.244419    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.246221 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.246203    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471b1d22-1296-4f99-aff8-a9a1c0e4604e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.246479 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.246462    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471b1d22-1296-4f99-aff8-a9a1c0e4604e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.252543 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.252518    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgtzn\" (UniqueName: \"kubernetes.io/projected/471b1d22-1296-4f99-aff8-a9a1c0e4604e-kube-api-access-qgtzn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk\" (UID: \"471b1d22-1296-4f99-aff8-a9a1c0e4604e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.361797 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.361745    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:05.630828 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.630798    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"]
Sep 16 05:01:05.634680 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:01:05.634642    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471b1d22_1296_4f99_aff8_a9a1c0e4604e.slice/crio-6f1d02cde3416f2dac42fb36a6a0e2d2e787d17bb55a8c48ed56ce2268159225 WatchSource:0}: Error finding container 6f1d02cde3416f2dac42fb36a6a0e2d2e787d17bb55a8c48ed56ce2268159225: Status 404 returned error can't find the container with id 6f1d02cde3416f2dac42fb36a6a0e2d2e787d17bb55a8c48ed56ce2268159225
Sep 16 05:01:05.733211 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:05.733180    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk" event={"ID":"471b1d22-1296-4f99-aff8-a9a1c0e4604e","Type":"ContainerStarted","Data":"6f1d02cde3416f2dac42fb36a6a0e2d2e787d17bb55a8c48ed56ce2268159225"}
Sep 16 05:01:06.480520 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:06.480490    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:01:06.739528 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:06.739449    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk" event={"ID":"471b1d22-1296-4f99-aff8-a9a1c0e4604e","Type":"ContainerStarted","Data":"62a3f7e0907134c1486f1e2ea8d2315f41e44814615a02d858f07f1aaf6dace0"}
Sep 16 05:01:08.175150 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:08.175120    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:01:13.391798 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:13.391744    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk" podStartSLOduration=8.218165111 podStartE2EDuration="8.391726746s" totalImagesPullingTime="173.561635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:01:06.759821372 +0000 UTC m=+730.425750985" watchObservedRunningTime="2026-09-16 05:01:13.391726746 +0000 UTC m=+737.057656338"
Sep 16 05:01:13.392622 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:13.392589    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:01:15.362738 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:15.362709    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:15.363114 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:15.362762    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:15.374996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:15.374973    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:15.790325 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:15.790299    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk"
Sep 16 05:01:23.228469 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:23.228436    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:01:25.281473 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:01:25.281433    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:02:01.710566 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:01.710529    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:05.254708 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:05.254672    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:05.342270 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:05.342235    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:32.000437 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:32.000351    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:32.098807 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:32.098775    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:33.080162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:33.080131    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:33.177755 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:33.177727    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:38.828245 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:38.828212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:48.309522 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:48.309487    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:48.410391 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:48.410354    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:48.510155 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:48.510121    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:48.710007 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:48.709974    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:48.814877 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:48.814840    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:48.908946 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:48.908912    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:49.140211 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:49.140180    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:49.240832 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:49.240793    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:49.739976 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:49.739943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:49.832335 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:49.832284    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:50.216988 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:50.216952    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:51.645403 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:51.645370    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:51.720771 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:51.720740    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:51.799317 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:51.799283    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:52.083314 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:52.083238    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:02:52.975510 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:52.975480    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"]
Sep 16 05:02:52.978105 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:52.978087    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:52.980811 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:52.980784    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 05:02:52.981347 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:52.981331    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 05:02:52.981400 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:52.981331    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c05179-serving-cert\""
Sep 16 05:02:52.981400 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:52.981379    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 05:02:52.984792 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:52.984767    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qdjvt\""
Sep 16 05:02:53.000752 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.000726    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"]
Sep 16 05:02:53.040433 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.040401    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:53.122204 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.122163    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rbr\" (UniqueName: \"kubernetes.io/projected/3f5eaacb-14c9-42e2-beea-69e3afb989ee-kube-api-access-s9rbr\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.122379 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.122216    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-metrics-certs\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.122379 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.122294    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5eaacb-14c9-42e2-beea-69e3afb989ee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.122379 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.122365    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-maas-api-tls\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.143551 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.143524    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:53.223551 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.223516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rbr\" (UniqueName: \"kubernetes.io/projected/3f5eaacb-14c9-42e2-beea-69e3afb989ee-kube-api-access-s9rbr\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.223763 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.223564    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-metrics-certs\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.223763 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.223607    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5eaacb-14c9-42e2-beea-69e3afb989ee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.223763 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.223661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-maas-api-tls\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.224102 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:02:53.223810    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-serving-cert: secret "maas-api-e2e-inf-a-c05179-serving-cert" not found
Sep 16 05:02:53.224102 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:02:53.223887    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-maas-api-tls podName:3f5eaacb-14c9-42e2-beea-69e3afb989ee nodeName:}" failed. No retries permitted until 2026-09-16 05:02:53.723866209 +0000 UTC m=+837.389795798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-maas-api-tls") pod "maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h" (UID: "3f5eaacb-14c9-42e2-beea-69e3afb989ee") : secret "maas-api-e2e-inf-a-c05179-serving-cert" not found
Sep 16 05:02:53.224453 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.224428    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5eaacb-14c9-42e2-beea-69e3afb989ee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.226303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.226243    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-metrics-certs\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.232090 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.232067    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rbr\" (UniqueName: \"kubernetes.io/projected/3f5eaacb-14c9-42e2-beea-69e3afb989ee-kube-api-access-s9rbr\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.495533 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.495442    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:53.649864 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.649829    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:53.727535 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.727486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-maas-api-tls\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.729972 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.729950    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-maas-api-tls\") pod \"maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.746317 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.746258    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:53.784614 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.784570    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"]
Sep 16 05:02:53.786950 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.786933    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:02:53.789077 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.789056    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c05179\""
Sep 16 05:02:53.789077 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.789074    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c05179-dockercfg-xktm5\""
Sep 16 05:02:53.796757 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.796737    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"]
Sep 16 05:02:53.888515 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.888478    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:53.929513 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.929477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmmf\" (UniqueName: \"kubernetes.io/projected/b7c5e637-bb87-4301-9b70-65b8c3953dd0-kube-api-access-rnmmf\") pod \"payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5\" (UID: \"b7c5e637-bb87-4301-9b70-65b8c3953dd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:02:53.929712 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:53.929548    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c5e637-bb87-4301-9b70-65b8c3953dd0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5\" (UID: \"b7c5e637-bb87-4301-9b70-65b8c3953dd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:02:54.009591 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.009505    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"]
Sep 16 05:02:54.012986 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.012960    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.014816 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.014790    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-12b6c6-serving-cert\""
Sep 16 05:02:54.023640 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.023617    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"]
Sep 16 05:02:54.030510 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.030479    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmmf\" (UniqueName: \"kubernetes.io/projected/b7c5e637-bb87-4301-9b70-65b8c3953dd0-kube-api-access-rnmmf\") pod \"payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5\" (UID: \"b7c5e637-bb87-4301-9b70-65b8c3953dd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:02:54.030658 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.030548    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c5e637-bb87-4301-9b70-65b8c3953dd0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5\" (UID: \"b7c5e637-bb87-4301-9b70-65b8c3953dd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:02:54.031422 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.031401    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c5e637-bb87-4301-9b70-65b8c3953dd0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5\" (UID: \"b7c5e637-bb87-4301-9b70-65b8c3953dd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:02:54.041453 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.041423    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmmf\" (UniqueName: \"kubernetes.io/projected/b7c5e637-bb87-4301-9b70-65b8c3953dd0-kube-api-access-rnmmf\") pod \"payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5\" (UID: \"b7c5e637-bb87-4301-9b70-65b8c3953dd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:02:54.047577 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.047554    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:54.097788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.097747    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:02:54.131733 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.131700    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nns77\" (UniqueName: \"kubernetes.io/projected/a5ea37d0-39e0-42ee-af77-6c233110236d-kube-api-access-nns77\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.131897 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.131754    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ea37d0-39e0-42ee-af77-6c233110236d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.131897 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.131797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.131897 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.131838    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-metrics-certs\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.145340 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.145303    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:54.159751 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.159722    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"]
Sep 16 05:02:54.164293 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:02:54.164256    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5eaacb_14c9_42e2_beea_69e3afb989ee.slice/crio-869d1f963324a4b4da5ae21646fb5a90fe682d749c4513e04a3c8db8853df930 WatchSource:0}: Error finding container 869d1f963324a4b4da5ae21646fb5a90fe682d749c4513e04a3c8db8853df930: Status 404 returned error can't find the container with id 869d1f963324a4b4da5ae21646fb5a90fe682d749c4513e04a3c8db8853df930
Sep 16 05:02:54.203181 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.203149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h" event={"ID":"3f5eaacb-14c9-42e2-beea-69e3afb989ee","Type":"ContainerStarted","Data":"869d1f963324a4b4da5ae21646fb5a90fe682d749c4513e04a3c8db8853df930"}
Sep 16 05:02:54.233172 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.233143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-metrics-certs\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.233350 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.233202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nns77\" (UniqueName: \"kubernetes.io/projected/a5ea37d0-39e0-42ee-af77-6c233110236d-kube-api-access-nns77\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.233350 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.233245    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ea37d0-39e0-42ee-af77-6c233110236d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.233350 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.233280    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.233513 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:02:54.233386    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-serving-cert: secret "maas-api-e2e-worker-w4-12b6c6-serving-cert" not found
Sep 16 05:02:54.233513 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:02:54.233453    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-maas-api-tls podName:a5ea37d0-39e0-42ee-af77-6c233110236d nodeName:}" failed. No retries permitted until 2026-09-16 05:02:54.733431182 +0000 UTC m=+838.399360755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-maas-api-tls") pod "maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5" (UID: "a5ea37d0-39e0-42ee-af77-6c233110236d") : secret "maas-api-e2e-worker-w4-12b6c6-serving-cert" not found
Sep 16 05:02:54.234040 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.234004    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ea37d0-39e0-42ee-af77-6c233110236d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.235981 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.235958    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-metrics-certs\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.241630 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.241586    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nns77\" (UniqueName: \"kubernetes.io/projected/a5ea37d0-39e0-42ee-af77-6c233110236d-kube-api-access-nns77\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.396241 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.396212    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"]
Sep 16 05:02:54.399838 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:02:54.399682    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c5e637_bb87_4301_9b70_65b8c3953dd0.slice/crio-ae6d3e2156a3b90918487e22d68e5af5d69ead2adcb5f3120def913989393ba6 WatchSource:0}: Error finding container ae6d3e2156a3b90918487e22d68e5af5d69ead2adcb5f3120def913989393ba6: Status 404 returned error can't find the container with id ae6d3e2156a3b90918487e22d68e5af5d69ead2adcb5f3120def913989393ba6
Sep 16 05:02:54.512111 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.512081    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:54.611842 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.611811    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:54.737009 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.736921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.739541 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.739511    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:54.925347 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:54.925301    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:55.022083 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:55.022050    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:55.210566 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:55.210448    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5" event={"ID":"b7c5e637-bb87-4301-9b70-65b8c3953dd0","Type":"ContainerStarted","Data":"ae6d3e2156a3b90918487e22d68e5af5d69ead2adcb5f3120def913989393ba6"}
Sep 16 05:02:55.215471 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:55.215381    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"]
Sep 16 05:02:55.768406 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:55.768368    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:55.848575 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:55.848544    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:56.217164 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:56.217130    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5" event={"ID":"a5ea37d0-39e0-42ee-af77-6c233110236d","Type":"ContainerStarted","Data":"3c5283cd9f7c14a87fffab489fd9201fe4083c9f5b93e6176086910b1ac784f8"}
Sep 16 05:02:56.728450 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:56.728415    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:56.819192 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:56.819148    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:56.919317 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:56.919277    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:57.026418 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.026385    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:57.123219 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.123106    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:57.181402 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.181365    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:57.223780 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.223745    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h" event={"ID":"3f5eaacb-14c9-42e2-beea-69e3afb989ee","Type":"ContainerStarted","Data":"69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7"}
Sep 16 05:02:57.224238 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.223854    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:02:57.225572 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.225547    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5" event={"ID":"b7c5e637-bb87-4301-9b70-65b8c3953dd0","Type":"ContainerStarted","Data":"0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99"}
Sep 16 05:02:57.225733 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.225633    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:02:57.227129 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.227110    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5" event={"ID":"a5ea37d0-39e0-42ee-af77-6c233110236d","Type":"ContainerStarted","Data":"80694f2b4d4825cd90aac4b5e849749ad2928c340e0c8a8661c87c66a25be5fa"}
Sep 16 05:02:57.227245 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.227232    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:02:57.241745 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.241672    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h" podStartSLOduration=2.440620085 podStartE2EDuration="5.241659125s" totalImagesPullingTime="2.80103904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:02:57.240092185 +0000 UTC m=+840.906021806" watchObservedRunningTime="2026-09-16 05:02:57.241659125 +0000 UTC m=+840.907588723"
Sep 16 05:02:57.259715 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.259672    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5" podStartSLOduration=1.752754954 podStartE2EDuration="4.259657454s" totalImagesPullingTime="2.5069025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:02:57.257536375 +0000 UTC m=+840.923465968" watchObservedRunningTime="2026-09-16 05:02:57.259657454 +0000 UTC m=+840.925587046"
Sep 16 05:02:57.276371 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.276331    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5" podStartSLOduration=2.5299569699999997 podStartE2EDuration="4.276307769s" totalImagesPullingTime="1.746350799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:02:57.273942583 +0000 UTC m=+840.939872194" watchObservedRunningTime="2026-09-16 05:02:57.276307769 +0000 UTC m=+840.942237394"
Sep 16 05:02:57.419649 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.419591    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:57.602112 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.602039    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"]
Sep 16 05:02:57.605397 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.605377    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.607326 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.607305    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ee7a60-serving-cert\""
Sep 16 05:02:57.618179 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.618097    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"]
Sep 16 05:02:57.640890 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.640859    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:57.740385 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.740359    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:57.765427 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.765403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19c8295-d377-4625-b588-942f7214ab31-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.765515 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.765457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.765563 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.765525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-metrics-certs\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.765563 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.765549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t84rx\" (UniqueName: \"kubernetes.io/projected/b19c8295-d377-4625-b588-942f7214ab31-kube-api-access-t84rx\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.866252 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.866179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19c8295-d377-4625-b588-942f7214ab31-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.866252 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.866238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.866430 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.866275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-metrics-certs\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.866430 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.866297    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t84rx\" (UniqueName: \"kubernetes.io/projected/b19c8295-d377-4625-b588-942f7214ab31-kube-api-access-t84rx\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.866430 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:02:57.866390    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-serving-cert: secret "maas-api-e2e-worker-w0-ee7a60-serving-cert" not found
Sep 16 05:02:57.866678 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:02:57.866457    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-maas-api-tls podName:b19c8295-d377-4625-b588-942f7214ab31 nodeName:}" failed. No retries permitted until 2026-09-16 05:02:58.366436618 +0000 UTC m=+842.032366196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-maas-api-tls") pod "maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c" (UID: "b19c8295-d377-4625-b588-942f7214ab31") : secret "maas-api-e2e-worker-w0-ee7a60-serving-cert" not found
Sep 16 05:02:57.866984 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.866965    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19c8295-d377-4625-b588-942f7214ab31-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.868886 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.868865    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-metrics-certs\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.874875 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.874847    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t84rx\" (UniqueName: \"kubernetes.io/projected/b19c8295-d377-4625-b588-942f7214ab31-kube-api-access-t84rx\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:57.958133 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:57.958079    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:58.052156 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:58.052132    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:58.151096 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:58.151070    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:58.235116 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:58.235080    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:02:58.371478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:58.371064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:58.373473 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:58.373454    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:58.527208 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:58.527146    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:02:58.797757 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:58.797632    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"]
Sep 16 05:02:58.800214 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:02:58.800185    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19c8295_d377_4625_b588_942f7214ab31.slice/crio-2daefd82430b52bf5e9332887581992add4a09f35493c1e7bf5115c410ca85cd WatchSource:0}: Error finding container 2daefd82430b52bf5e9332887581992add4a09f35493c1e7bf5115c410ca85cd: Status 404 returned error can't find the container with id 2daefd82430b52bf5e9332887581992add4a09f35493c1e7bf5115c410ca85cd
Sep 16 05:02:59.240684 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:59.240612    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c" event={"ID":"b19c8295-d377-4625-b588-942f7214ab31","Type":"ContainerStarted","Data":"2daefd82430b52bf5e9332887581992add4a09f35493c1e7bf5115c410ca85cd"}
Sep 16 05:02:59.453708 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:02:59.453635    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:00.092695 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:00.092661    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:00.182776 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:00.182747    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:00.246575 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:00.246548    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c" event={"ID":"b19c8295-d377-4625-b588-942f7214ab31","Type":"ContainerStarted","Data":"d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3"}
Sep 16 05:03:00.246914 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:00.246657    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:03:00.264500 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:00.264448    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c" podStartSLOduration=2.886761385 podStartE2EDuration="3.26442954s" totalImagesPullingTime="377.668155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:03:00.263414614 +0000 UTC m=+843.929344220" watchObservedRunningTime="2026-09-16 05:03:00.26442954 +0000 UTC m=+843.930359134"
Sep 16 05:03:00.554113 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:00.554087    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:00.756172 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:00.756140    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:00.843310 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:00.843247    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:01.749314 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:01.749281    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:02.732989 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:02.732950    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:03.237799 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.237774    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:03:03.238307 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.238262    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:03:03.372486 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.372452    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:03.459650 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.459621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:03.813880 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.813848    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:03.913908 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.913875    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:03.951012 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.950983    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"]
Sep 16 05:03:03.954452 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.954435    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:03:03.956396 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.956378    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-d05eb5\""
Sep 16 05:03:03.956484 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.956452    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-d05eb5-dockercfg-s2kcf\""
Sep 16 05:03:03.964655 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.964635    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"]
Sep 16 05:03:03.993849 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:03.993828    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:04.119328 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.119264    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09b053c-30cc-4292-b4ad-a9064e672a66-plugins-config-volume\") pod \"payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7\" (UID: \"a09b053c-30cc-4292-b4ad-a9064e672a66\") " pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:03:04.119446 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.119345    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jfn\" (UniqueName: \"kubernetes.io/projected/a09b053c-30cc-4292-b4ad-a9064e672a66-kube-api-access-68jfn\") pod \"payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7\" (UID: \"a09b053c-30cc-4292-b4ad-a9064e672a66\") " pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:03:04.220097 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.220071    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09b053c-30cc-4292-b4ad-a9064e672a66-plugins-config-volume\") pod \"payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7\" (UID: \"a09b053c-30cc-4292-b4ad-a9064e672a66\") " pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:03:04.220232 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.220157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jfn\" (UniqueName: \"kubernetes.io/projected/a09b053c-30cc-4292-b4ad-a9064e672a66-kube-api-access-68jfn\") pod \"payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7\" (UID: \"a09b053c-30cc-4292-b4ad-a9064e672a66\") " pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:03:04.220764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.220744    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09b053c-30cc-4292-b4ad-a9064e672a66-plugins-config-volume\") pod \"payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7\" (UID: \"a09b053c-30cc-4292-b4ad-a9064e672a66\") " pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:03:04.228725 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.228706    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jfn\" (UniqueName: \"kubernetes.io/projected/a09b053c-30cc-4292-b4ad-a9064e672a66-kube-api-access-68jfn\") pod \"payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7\" (UID: \"a09b053c-30cc-4292-b4ad-a9064e672a66\") " pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:03:04.265212 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.265187    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:03:04.355888 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.355861    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:04.419832 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.419807    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:04.540332 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.540297    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"]
Sep 16 05:03:04.543405 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:03:04.543383    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09b053c_30cc_4292_b4ad_a9064e672a66.slice/crio-0abb981045c1f208467a4465077c645a110a6382c56ed5491a94aedb8be7fb9d WatchSource:0}: Error finding container 0abb981045c1f208467a4465077c645a110a6382c56ed5491a94aedb8be7fb9d: Status 404 returned error can't find the container with id 0abb981045c1f208467a4465077c645a110a6382c56ed5491a94aedb8be7fb9d
Sep 16 05:03:04.570697 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.570677    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:04.915570 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:04.915541    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:05.009865 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:05.009835    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:05.269490 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:05.269416    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7" event={"ID":"a09b053c-30cc-4292-b4ad-a9064e672a66","Type":"ContainerStarted","Data":"fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5"}
Sep 16 05:03:05.269490 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:05.269462    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7" event={"ID":"a09b053c-30cc-4292-b4ad-a9064e672a66","Type":"ContainerStarted","Data":"0abb981045c1f208467a4465077c645a110a6382c56ed5491a94aedb8be7fb9d"}
Sep 16 05:03:05.269881 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:05.269554    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:03:05.284914 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:05.284875    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7" podStartSLOduration=2.284861487 podStartE2EDuration="2.284861487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:03:05.282850769 +0000 UTC m=+848.948780373" watchObservedRunningTime="2026-09-16 05:03:05.284861487 +0000 UTC m=+848.950791080"
Sep 16 05:03:05.337396 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:05.337371    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:05.970730 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:05.970701    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:06.256666 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:06.256586    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:03:06.811749 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:06.811719    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:06.906343 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:06.906313    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:07.009570 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:07.009547    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:07.106627 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:07.106536    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:07.212583 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:07.212563    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:07.740508 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:07.740476    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:08.233696 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:08.233656    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:03:08.542420 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:08.542326    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:08.681777 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:08.681736    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:03:09.644603 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:09.644560    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:09.871533 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:09.871496    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:10.387000 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:10.386969    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:11.277163 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:11.277130    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:11.465120 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:11.465090    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:11.567175 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:11.567106    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:11.793526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:11.793495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:11.935046 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:11.935007    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:12.052700 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:12.052659    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:12.146801 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:12.146773    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:12.251871 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:12.251796    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:13.226783 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:13.226742    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:13.435003 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:13.434955    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:14.435555 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:14.435523    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:15.329368 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:15.329341    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:15.458747 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:15.458717    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:15.552898 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:15.552870    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:15.802778 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:15.802747    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:15.910725 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:15.910701    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:16.001457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:16.001429    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:16.099920 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:16.099848    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:16.203763 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:16.203736    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:16.276458 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:16.276435    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:03:16.478851 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:16.478823    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:17.363784 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:17.363756    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:18.213185 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:18.213155    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:18.881229 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:18.881193    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:19.603228 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.603198    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"]
Sep 16 05:03:19.608812 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.608787    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.610657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.610639    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d1f291-serving-cert\""
Sep 16 05:03:19.617875 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.617853    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"]
Sep 16 05:03:19.636736 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.636714    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:19.693714 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.693693    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:19.739927 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.739905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-maas-api-tls\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.740048 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.739944    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146993cb-e432-4446-94a4-75e95c8c3ad3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.740048 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.739965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9crn7\" (UniqueName: \"kubernetes.io/projected/146993cb-e432-4446-94a4-75e95c8c3ad3-kube-api-access-9crn7\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.740163 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.740113    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-metrics-certs\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.841277 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.841250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-maas-api-tls\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.841385 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.841281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146993cb-e432-4446-94a4-75e95c8c3ad3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.841385 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.841308    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9crn7\" (UniqueName: \"kubernetes.io/projected/146993cb-e432-4446-94a4-75e95c8c3ad3-kube-api-access-9crn7\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.841385 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.841380    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-metrics-certs\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.841542 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:03:19.841404    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-serving-cert: secret "maas-api-e2e-inf-b-d1f291-serving-cert" not found
Sep 16 05:03:19.841542 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:03:19.841475    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-maas-api-tls podName:146993cb-e432-4446-94a4-75e95c8c3ad3 nodeName:}" failed. No retries permitted until 2026-09-16 05:03:20.341455855 +0000 UTC m=+864.007385446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-maas-api-tls") pod "maas-api-e2e-inf-b-d1f291-685b58794f-99js4" (UID: "146993cb-e432-4446-94a4-75e95c8c3ad3") : secret "maas-api-e2e-inf-b-d1f291-serving-cert" not found
Sep 16 05:03:19.842011 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.841990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146993cb-e432-4446-94a4-75e95c8c3ad3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.843710 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.843686    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-metrics-certs\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.852332 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.852291    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9crn7\" (UniqueName: \"kubernetes.io/projected/146993cb-e432-4446-94a4-75e95c8c3ad3-kube-api-access-9crn7\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:19.985962 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:19.985932    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:20.091658 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.091621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:20.183267 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.183239    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:20.186413 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.186389    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"]
Sep 16 05:03:20.229899 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.229872    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"]
Sep 16 05:03:20.230017 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.229953    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:03:20.232263 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.232243    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d1f291-dockercfg-gfhw9\""
Sep 16 05:03:20.232473 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.232456    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d1f291\""
Sep 16 05:03:20.344913 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.344879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-maas-api-tls\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:20.345074 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.344935    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgvr7\" (UniqueName: \"kubernetes.io/projected/fdeb454c-1e72-4c98-add6-f2dc16469300-kube-api-access-bgvr7\") pod \"payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll\" (UID: \"fdeb454c-1e72-4c98-add6-f2dc16469300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:03:20.345074 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.345010    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdeb454c-1e72-4c98-add6-f2dc16469300-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll\" (UID: \"fdeb454c-1e72-4c98-add6-f2dc16469300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:03:20.347274 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.347255    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-maas-api-tls\") pod \"maas-api-e2e-inf-b-d1f291-685b58794f-99js4\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:20.445446 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.445422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdeb454c-1e72-4c98-add6-f2dc16469300-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll\" (UID: \"fdeb454c-1e72-4c98-add6-f2dc16469300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:03:20.445560 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.445478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgvr7\" (UniqueName: \"kubernetes.io/projected/fdeb454c-1e72-4c98-add6-f2dc16469300-kube-api-access-bgvr7\") pod \"payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll\" (UID: \"fdeb454c-1e72-4c98-add6-f2dc16469300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:03:20.446075 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.446058    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdeb454c-1e72-4c98-add6-f2dc16469300-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll\" (UID: \"fdeb454c-1e72-4c98-add6-f2dc16469300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:03:20.454977 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.454959    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgvr7\" (UniqueName: \"kubernetes.io/projected/fdeb454c-1e72-4c98-add6-f2dc16469300-kube-api-access-bgvr7\") pod \"payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll\" (UID: \"fdeb454c-1e72-4c98-add6-f2dc16469300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:03:20.520757 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.520697    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:20.540561 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.540533    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:03:20.557403 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.557379    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:20.796174 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.796147    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"]
Sep 16 05:03:20.798049 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:03:20.798018    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146993cb_e432_4446_94a4_75e95c8c3ad3.slice/crio-2d012173df1ac2f749def4a6778beb2b46a47c1ae4daa6d7161238ef30fee0b2 WatchSource:0}: Error finding container 2d012173df1ac2f749def4a6778beb2b46a47c1ae4daa6d7161238ef30fee0b2: Status 404 returned error can't find the container with id 2d012173df1ac2f749def4a6778beb2b46a47c1ae4daa6d7161238ef30fee0b2
Sep 16 05:03:20.823708 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:20.823677    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"]
Sep 16 05:03:20.826351 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:03:20.826317    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdeb454c_1e72_4c98_add6_f2dc16469300.slice/crio-f245d6419f9dd41932cbe1660ef744ee94b7f30466b9a27fb0ae64cb2a99852e WatchSource:0}: Error finding container f245d6419f9dd41932cbe1660ef744ee94b7f30466b9a27fb0ae64cb2a99852e: Status 404 returned error can't find the container with id f245d6419f9dd41932cbe1660ef744ee94b7f30466b9a27fb0ae64cb2a99852e
Sep 16 05:03:21.347163 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:21.346963    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll" event={"ID":"fdeb454c-1e72-4c98-add6-f2dc16469300","Type":"ContainerStarted","Data":"1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80"}
Sep 16 05:03:21.347163 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:21.347043    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll" event={"ID":"fdeb454c-1e72-4c98-add6-f2dc16469300","Type":"ContainerStarted","Data":"f245d6419f9dd41932cbe1660ef744ee94b7f30466b9a27fb0ae64cb2a99852e"}
Sep 16 05:03:21.347163 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:21.347072    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:03:21.349145 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:21.349117    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4" event={"ID":"146993cb-e432-4446-94a4-75e95c8c3ad3","Type":"ContainerStarted","Data":"7e0e6e0867aefaf463d7eae0c9dad84d3960c6aec09e0a77a0d531a8c8090d09"}
Sep 16 05:03:21.349268 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:21.349165    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4" event={"ID":"146993cb-e432-4446-94a4-75e95c8c3ad3","Type":"ContainerStarted","Data":"2d012173df1ac2f749def4a6778beb2b46a47c1ae4daa6d7161238ef30fee0b2"}
Sep 16 05:03:21.349268 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:21.349193    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:21.400885 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:21.400797    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll" podStartSLOduration=1.400782833 podStartE2EDuration="1.400782833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:03:21.396849704 +0000 UTC m=+865.062779326" watchObservedRunningTime="2026-09-16 05:03:21.400782833 +0000 UTC m=+865.066712425"
Sep 16 05:03:21.426765 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:21.426725    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4" podStartSLOduration=2.087476336 podStartE2EDuration="2.426712655s" totalImagesPullingTime="339.236319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:03:21.424018108 +0000 UTC m=+865.089947720" watchObservedRunningTime="2026-09-16 05:03:21.426712655 +0000 UTC m=+865.092642248"
Sep 16 05:03:22.057169 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:22.057135    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:22.326864 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:22.326798    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:22.437221 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:22.437196    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:22.527888 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:22.527862    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:22.651071 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:22.651046    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:23.123342 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:23.123256    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:23.985040 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:23.985012    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:24.923944 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:24.923914    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:25.103850 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:25.103819    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:25.205836 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:25.205766    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:25.486499 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:25.486431    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:03:25.880397 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:25.880367    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:27.044549 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:27.044523    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:27.359828 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:27.359756    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:03:27.739688 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:27.739660    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:28.409701 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:28.409672    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:28.842738 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:28.842669    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:28.954267 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:28.954236    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:31.038822 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:31.038789    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:31.114786 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:31.114761    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:31.187110 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:31.187085    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:31.249484 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:31.249454    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:31.364996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:31.364917    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:31.637972 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:31.637942    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:31.815146 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:31.815107    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:31.912653 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:31.912552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:32.010693 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:32.010658    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:32.354766 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:32.354680    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:32.356009 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:32.355987    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:03:32.510162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:32.510134    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:32.607880 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:32.607784    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:33.264309 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:33.264271    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:34.032951 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:34.032920    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:34.636972 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:34.636937    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:34.738126 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:34.738098    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:34.838303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:34.838236    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:35.564586 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:35.564555    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:36.308818 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.308786    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"]
Sep 16 05:03:36.313343 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.313325    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.315360 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.315340    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-70554e75-serving-cert\""
Sep 16 05:03:36.321336 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.321311    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"]
Sep 16 05:03:36.373059 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.373032    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-metrics-certs\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.373192 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.373089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7479e061-9704-4a68-832a-43c3761cafe6-openshift-service-ca\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.373192 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.373159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-maas-api-tls\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.373295 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.373230    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v47cm\" (UniqueName: \"kubernetes.io/projected/7479e061-9704-4a68-832a-43c3761cafe6-kube-api-access-v47cm\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.400580 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.400559    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:36.474478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.474458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7479e061-9704-4a68-832a-43c3761cafe6-openshift-service-ca\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.474624 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.474495    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-maas-api-tls\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.474624 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.474525    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47cm\" (UniqueName: \"kubernetes.io/projected/7479e061-9704-4a68-832a-43c3761cafe6-kube-api-access-v47cm\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.474624 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.474619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-metrics-certs\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.474795 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:03:36.474661    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-serving-cert: secret "maas-api-e2e-migrate-70554e75-serving-cert" not found
Sep 16 05:03:36.474795 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:03:36.474762    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-maas-api-tls podName:7479e061-9704-4a68-832a-43c3761cafe6 nodeName:}" failed. No retries permitted until 2026-09-16 05:03:36.974740254 +0000 UTC m=+880.640669843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-maas-api-tls") pod "maas-api-e2e-migrate-70554e75-67466cb979-5j69v" (UID: "7479e061-9704-4a68-832a-43c3761cafe6") : secret "maas-api-e2e-migrate-70554e75-serving-cert" not found
Sep 16 05:03:36.475207 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.475187    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7479e061-9704-4a68-832a-43c3761cafe6-openshift-service-ca\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.477682 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.477659    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-metrics-certs\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.484057 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.484033    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v47cm\" (UniqueName: \"kubernetes.io/projected/7479e061-9704-4a68-832a-43c3761cafe6-kube-api-access-v47cm\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.502455 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.502428    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:36.631260 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.631241    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:36.859102 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.859066    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:36.964725 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.964651    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:36.980013 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.979981    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-maas-api-tls\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.982434 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.982412    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-maas-api-tls\") pod \"maas-api-e2e-migrate-70554e75-67466cb979-5j69v\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:36.993789 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.993768    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"]
Sep 16 05:03:36.997231 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.997216    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:03:36.999182 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.999166    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-70554e75-dockercfg-dqc48\""
Sep 16 05:03:36.999240 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:36.999194    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-70554e75\""
Sep 16 05:03:37.007863 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.007843    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"]
Sep 16 05:03:37.080583 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.080551    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw6qv\" (UniqueName: \"kubernetes.io/projected/7fc4df25-306c-4ca2-b278-4192997ce009-kube-api-access-nw6qv\") pod \"payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl\" (UID: \"7fc4df25-306c-4ca2-b278-4192997ce009\") " pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:03:37.080744 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.080687    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc4df25-306c-4ca2-b278-4192997ce009-plugins-config-volume\") pod \"payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl\" (UID: \"7fc4df25-306c-4ca2-b278-4192997ce009\") " pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:03:37.179214 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.179189    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:03:37.181889 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.181867    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc4df25-306c-4ca2-b278-4192997ce009-plugins-config-volume\") pod \"payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl\" (UID: \"7fc4df25-306c-4ca2-b278-4192997ce009\") " pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:03:37.182001 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.181914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw6qv\" (UniqueName: \"kubernetes.io/projected/7fc4df25-306c-4ca2-b278-4192997ce009-kube-api-access-nw6qv\") pod \"payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl\" (UID: \"7fc4df25-306c-4ca2-b278-4192997ce009\") " pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:03:37.182736 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.182718    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc4df25-306c-4ca2-b278-4192997ce009-plugins-config-volume\") pod \"payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl\" (UID: \"7fc4df25-306c-4ca2-b278-4192997ce009\") " pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:03:37.189425 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.189406    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw6qv\" (UniqueName: \"kubernetes.io/projected/7fc4df25-306c-4ca2-b278-4192997ce009-kube-api-access-nw6qv\") pod \"payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl\" (UID: \"7fc4df25-306c-4ca2-b278-4192997ce009\") " pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:03:37.225631 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.225568    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:37.248389 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.248358    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:37.308637 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.308562    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:03:37.496305 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.496268    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"]
Sep 16 05:03:37.498519 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:03:37.498493    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7479e061_9704_4a68_832a_43c3761cafe6.slice/crio-a77903b8c384ed6037673684a41df053c080d95da26127488ec61e0b78354f5a WatchSource:0}: Error finding container a77903b8c384ed6037673684a41df053c080d95da26127488ec61e0b78354f5a: Status 404 returned error can't find the container with id a77903b8c384ed6037673684a41df053c080d95da26127488ec61e0b78354f5a
Sep 16 05:03:37.584038 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:37.584018    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"]
Sep 16 05:03:37.586038 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:03:37.586017    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc4df25_306c_4ca2_b278_4192997ce009.slice/crio-c2dbb774c798e03735bf25be9094c6a95e12af3ebca51dd8eb1829643fafa52f WatchSource:0}: Error finding container c2dbb774c798e03735bf25be9094c6a95e12af3ebca51dd8eb1829643fafa52f: Status 404 returned error can't find the container with id c2dbb774c798e03735bf25be9094c6a95e12af3ebca51dd8eb1829643fafa52f
Sep 16 05:03:38.226378 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:38.226309    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:38.434541 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:38.434503    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl" event={"ID":"7fc4df25-306c-4ca2-b278-4192997ce009","Type":"ContainerStarted","Data":"b74be9d826f5c11bd88167e0099d2c0d9f1cf42e5c29f83d288ffb0ba0041941"}
Sep 16 05:03:38.434734 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:38.434647    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:03:38.434734 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:38.434694    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl" event={"ID":"7fc4df25-306c-4ca2-b278-4192997ce009","Type":"ContainerStarted","Data":"c2dbb774c798e03735bf25be9094c6a95e12af3ebca51dd8eb1829643fafa52f"}
Sep 16 05:03:38.435973 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:38.435948    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v" event={"ID":"7479e061-9704-4a68-832a-43c3761cafe6","Type":"ContainerStarted","Data":"765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed"}
Sep 16 05:03:38.436075 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:38.435996    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:38.436075 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:38.436014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v" event={"ID":"7479e061-9704-4a68-832a-43c3761cafe6","Type":"ContainerStarted","Data":"a77903b8c384ed6037673684a41df053c080d95da26127488ec61e0b78354f5a"}
Sep 16 05:03:38.453987 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:38.453944    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl" podStartSLOduration=2.453931304 podStartE2EDuration="2.453931304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:03:38.452559998 +0000 UTC m=+882.118489617" watchObservedRunningTime="2026-09-16 05:03:38.453931304 +0000 UTC m=+882.119860897"
Sep 16 05:03:38.471209 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:38.471161    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v" podStartSLOduration=2.084375018 podStartE2EDuration="2.47114361s" totalImagesPullingTime="386.768592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:03:38.469402834 +0000 UTC m=+882.135332442" watchObservedRunningTime="2026-09-16 05:03:38.47114361 +0000 UTC m=+882.137073204"
Sep 16 05:03:38.847630 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:38.847576    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:39.542569 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:39.542543    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:40.343393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:40.343342    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:40.732370 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:40.732339    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:41.020209 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:41.020141    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:41.674879 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:41.674848    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:42.280417 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:42.280384    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:44.446582 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:44.446554    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:03:49.442937 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:49.442910    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:03:49.476795 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:49.476765    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:03:56.750348 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:56.750320    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:03:56.751270 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:56.751243    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:03:57.252893 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:57.252867    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:57.360315 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:57.360281    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:58.538477 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:58.538450    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:59.167907 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:59.167859    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:03:59.945536 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:03:59.945505    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:00.753028 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:00.752994    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:01.378147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:01.378114    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:02.027943 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:02.027912    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:02.732827 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:02.732797    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:03.340551 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:03.340518    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:03.985792 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:03.985766    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:04:04.041617 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:04.041557    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:04.806411 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:04.806385    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:04.906978 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:04.906951    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:05.764229 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:05.764197    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:06.285799 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.285766    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"]
Sep 16 05:04:06.286015 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.285991    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v" podUID="7479e061-9704-4a68-832a-43c3761cafe6" containerName="maas-api" containerID="cri-o://765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed" gracePeriod=30
Sep 16 05:04:06.315308 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.315267    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:06.330755 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.330729    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"]
Sep 16 05:04:06.331047 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.331021    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl" podUID="7fc4df25-306c-4ca2-b278-4192997ce009" containerName="payload-processing" containerID="cri-o://b74be9d826f5c11bd88167e0099d2c0d9f1cf42e5c29f83d288ffb0ba0041941" gracePeriod=30
Sep 16 05:04:06.411697 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.411668    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:06.493757 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.493725    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:06.564878 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.564697    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:06.575018 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.574992    2573 generic.go:356] "Generic (PLEG): container finished" podID="7fc4df25-306c-4ca2-b278-4192997ce009" containerID="b74be9d826f5c11bd88167e0099d2c0d9f1cf42e5c29f83d288ffb0ba0041941" exitCode=0
Sep 16 05:04:06.575167 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.575071    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl" event={"ID":"7fc4df25-306c-4ca2-b278-4192997ce009","Type":"ContainerDied","Data":"b74be9d826f5c11bd88167e0099d2c0d9f1cf42e5c29f83d288ffb0ba0041941"}
Sep 16 05:04:06.575845 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.575827    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:04:06.576460 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.576439    2573 generic.go:356] "Generic (PLEG): container finished" podID="7479e061-9704-4a68-832a-43c3761cafe6" containerID="765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed" exitCode=0
Sep 16 05:04:06.576549 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.576511    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v" event={"ID":"7479e061-9704-4a68-832a-43c3761cafe6","Type":"ContainerDied","Data":"765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed"}
Sep 16 05:04:06.576620 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.576573    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v" event={"ID":"7479e061-9704-4a68-832a-43c3761cafe6","Type":"ContainerDied","Data":"a77903b8c384ed6037673684a41df053c080d95da26127488ec61e0b78354f5a"}
Sep 16 05:04:06.576683 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.576646    2573 scope.go:117] "RemoveContainer" containerID="765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed"
Sep 16 05:04:06.587463 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.587445    2573 scope.go:117] "RemoveContainer" containerID="765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed"
Sep 16 05:04:06.587825 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:04:06.587806    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed\": container with ID starting with 765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed not found: ID does not exist" containerID="765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed"
Sep 16 05:04:06.587908 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.587837    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed"} err="failed to get container status \"765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed\": rpc error: code = NotFound desc = could not find container \"765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed\": container with ID starting with 765003c39882faa373f84b26dec0f95dcc4ad1c4058e1975100a9938157b95ed not found: ID does not exist"
Sep 16 05:04:06.624817 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.624540    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:04:06.648464 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.648433    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:06.709404 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.709381    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-maas-api-tls\") pod \"7479e061-9704-4a68-832a-43c3761cafe6\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") "
Sep 16 05:04:06.709530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.709414    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v47cm\" (UniqueName: \"kubernetes.io/projected/7479e061-9704-4a68-832a-43c3761cafe6-kube-api-access-v47cm\") pod \"7479e061-9704-4a68-832a-43c3761cafe6\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") "
Sep 16 05:04:06.709530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.709444    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7479e061-9704-4a68-832a-43c3761cafe6-openshift-service-ca\") pod \"7479e061-9704-4a68-832a-43c3761cafe6\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") "
Sep 16 05:04:06.709685 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.709643    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-metrics-certs\") pod \"7479e061-9704-4a68-832a-43c3761cafe6\" (UID: \"7479e061-9704-4a68-832a-43c3761cafe6\") "
Sep 16 05:04:06.709685 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.709672    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw6qv\" (UniqueName: \"kubernetes.io/projected/7fc4df25-306c-4ca2-b278-4192997ce009-kube-api-access-nw6qv\") pod \"7fc4df25-306c-4ca2-b278-4192997ce009\" (UID: \"7fc4df25-306c-4ca2-b278-4192997ce009\") "
Sep 16 05:04:06.709812 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.709723    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc4df25-306c-4ca2-b278-4192997ce009-plugins-config-volume\") pod \"7fc4df25-306c-4ca2-b278-4192997ce009\" (UID: \"7fc4df25-306c-4ca2-b278-4192997ce009\") "
Sep 16 05:04:06.709812 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.709738    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7479e061-9704-4a68-832a-43c3761cafe6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7479e061-9704-4a68-832a-43c3761cafe6" (UID: "7479e061-9704-4a68-832a-43c3761cafe6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:04:06.710043 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.709983    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7479e061-9704-4a68-832a-43c3761cafe6-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:06.710157 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.710142    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc4df25-306c-4ca2-b278-4192997ce009-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7fc4df25-306c-4ca2-b278-4192997ce009" (UID: "7fc4df25-306c-4ca2-b278-4192997ce009"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:04:06.712134 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.712119    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7479e061-9704-4a68-832a-43c3761cafe6" (UID: "7479e061-9704-4a68-832a-43c3761cafe6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:04:06.712209 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.712167    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7479e061-9704-4a68-832a-43c3761cafe6-kube-api-access-v47cm" (OuterVolumeSpecName: "kube-api-access-v47cm") pod "7479e061-9704-4a68-832a-43c3761cafe6" (UID: "7479e061-9704-4a68-832a-43c3761cafe6"). InnerVolumeSpecName "kube-api-access-v47cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:04:06.712263 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.712220    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7479e061-9704-4a68-832a-43c3761cafe6" (UID: "7479e061-9704-4a68-832a-43c3761cafe6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:04:06.712362 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.712254    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc4df25-306c-4ca2-b278-4192997ce009-kube-api-access-nw6qv" (OuterVolumeSpecName: "kube-api-access-nw6qv") pod "7fc4df25-306c-4ca2-b278-4192997ce009" (UID: "7fc4df25-306c-4ca2-b278-4192997ce009"). InnerVolumeSpecName "kube-api-access-nw6qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:04:06.810681 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.810629    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:06.810681 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.810650    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw6qv\" (UniqueName: \"kubernetes.io/projected/7fc4df25-306c-4ca2-b278-4192997ce009-kube-api-access-nw6qv\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:06.810681 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.810659    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc4df25-306c-4ca2-b278-4192997ce009-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:06.810681 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.810669    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479e061-9704-4a68-832a-43c3761cafe6-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:06.810681 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.810677    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v47cm\" (UniqueName: \"kubernetes.io/projected/7479e061-9704-4a68-832a-43c3761cafe6-kube-api-access-v47cm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:06.919475 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:06.919426    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:07.067913 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.067853    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:07.357800 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.357727    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:07.581794 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.581766    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"
Sep 16 05:04:07.581982 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.581770    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl" event={"ID":"7fc4df25-306c-4ca2-b278-4192997ce009","Type":"ContainerDied","Data":"c2dbb774c798e03735bf25be9094c6a95e12af3ebca51dd8eb1829643fafa52f"}
Sep 16 05:04:07.581982 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.581915    2573 scope.go:117] "RemoveContainer" containerID="b74be9d826f5c11bd88167e0099d2c0d9f1cf42e5c29f83d288ffb0ba0041941"
Sep 16 05:04:07.582751 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.582728    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"
Sep 16 05:04:07.602695 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.602673    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"]
Sep 16 05:04:07.606237 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.606217    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70554e75-67466cb979-5j69v"]
Sep 16 05:04:07.614210 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.614162    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"]
Sep 16 05:04:07.620105 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.620085    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70554e75-794bf49fcd-xhxjl"]
Sep 16 05:04:07.636567 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.636546    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:07.736930 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:07.736901    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:08.246363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:08.246336    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:08.785578 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:08.785549    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7479e061-9704-4a68-832a-43c3761cafe6" path="/var/lib/kubelet/pods/7479e061-9704-4a68-832a-43c3761cafe6/volumes"
Sep 16 05:04:08.785917 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:08.785903    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fc4df25-306c-4ca2-b278-4192997ce009" path="/var/lib/kubelet/pods/7fc4df25-306c-4ca2-b278-4192997ce009/volumes"
Sep 16 05:04:09.144327 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:09.144302    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:09.665537 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:09.665498    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:09.755319 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:09.755281    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:10.003940 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:10.003857    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:10.717845 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:10.717816    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:11.515701 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.515675    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"]
Sep 16 05:04:11.516042 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.516029    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc4df25-306c-4ca2-b278-4192997ce009" containerName="payload-processing"
Sep 16 05:04:11.516086 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.516044    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc4df25-306c-4ca2-b278-4192997ce009" containerName="payload-processing"
Sep 16 05:04:11.516086 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.516064    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7479e061-9704-4a68-832a-43c3761cafe6" containerName="maas-api"
Sep 16 05:04:11.516086 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.516069    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7479e061-9704-4a68-832a-43c3761cafe6" containerName="maas-api"
Sep 16 05:04:11.516191 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.516126    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fc4df25-306c-4ca2-b278-4192997ce009" containerName="payload-processing"
Sep 16 05:04:11.516191 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.516136    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7479e061-9704-4a68-832a-43c3761cafe6" containerName="maas-api"
Sep 16 05:04:11.520857 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.520834    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.522924 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.522899    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-25f3e8f3-serving-cert\""
Sep 16 05:04:11.531587 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.531564    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"]
Sep 16 05:04:11.554135 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.554112    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:11.647475 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.647455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-maas-api-tls\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.647564 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.647482    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwpn\" (UniqueName: \"kubernetes.io/projected/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-kube-api-access-6dwpn\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.647564 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.647520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-openshift-service-ca\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.647657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.647590    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-metrics-certs\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.654589 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.654572    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:11.748475 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.748448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-maas-api-tls\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.748475 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.748475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwpn\" (UniqueName: \"kubernetes.io/projected/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-kube-api-access-6dwpn\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.748841 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.748529    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-openshift-service-ca\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.748841 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.748576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-metrics-certs\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.749245 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.749224    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-openshift-service-ca\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.750949 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.750924    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-metrics-certs\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.751030 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.750969    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-maas-api-tls\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.754357 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.754336    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:11.755750 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.755731    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwpn\" (UniqueName: \"kubernetes.io/projected/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-kube-api-access-6dwpn\") pod \"maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:11.832634 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:11.832561    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:12.003202 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.003168    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:12.101588 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.101510    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:12.111779 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.111750    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"]
Sep 16 05:04:12.114706 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:04:12.114681    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69c08a6_9ae5_4dad_9ee8_446bc54088c0.slice/crio-3ecd691f68b630df682ff64e8896d96489d25e09f5ab496845662d6b3fefca40 WatchSource:0}: Error finding container 3ecd691f68b630df682ff64e8896d96489d25e09f5ab496845662d6b3fefca40: Status 404 returned error can't find the container with id 3ecd691f68b630df682ff64e8896d96489d25e09f5ab496845662d6b3fefca40
Sep 16 05:04:12.170748 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.170725    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"]
Sep 16 05:04:12.175065 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.175049    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:12.177011 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.176985    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-25f3e8f3\""
Sep 16 05:04:12.177088 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.177064    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-25f3e8f3-dockercfg-nwd7z\""
Sep 16 05:04:12.183929 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.183908    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"]
Sep 16 05:04:12.354371 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.354276    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwt5f\" (UniqueName: \"kubernetes.io/projected/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-kube-api-access-lwt5f\") pod \"payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc\" (UID: \"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:12.354528 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.354405    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc\" (UID: \"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:12.417379 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.417351    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:12.455564 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.455533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwt5f\" (UniqueName: \"kubernetes.io/projected/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-kube-api-access-lwt5f\") pod \"payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc\" (UID: \"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:12.455687 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.455641    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc\" (UID: \"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:12.456386 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.456368    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc\" (UID: \"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:12.463233 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.463216    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwt5f\" (UniqueName: \"kubernetes.io/projected/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-kube-api-access-lwt5f\") pod \"payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc\" (UID: \"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:12.486186 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.486164    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:12.610768 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.610705    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc" event={"ID":"c69c08a6-9ae5-4dad-9ee8-446bc54088c0","Type":"ContainerStarted","Data":"3ecd691f68b630df682ff64e8896d96489d25e09f5ab496845662d6b3fefca40"}
Sep 16 05:04:12.756549 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:12.756521    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"]
Sep 16 05:04:12.759836 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:04:12.759808    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8760f52c_c6d5_4c2f_84ed_35d04b0f35f7.slice/crio-82f4684c43d6c92d58f76a9c142bb4329fe70deb5395b99b6f5e0aa56c4594cd WatchSource:0}: Error finding container 82f4684c43d6c92d58f76a9c142bb4329fe70deb5395b99b6f5e0aa56c4594cd: Status 404 returned error can't find the container with id 82f4684c43d6c92d58f76a9c142bb4329fe70deb5395b99b6f5e0aa56c4594cd
Sep 16 05:04:13.114234 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:13.114209    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:13.616937 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:13.616906    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc" event={"ID":"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7","Type":"ContainerStarted","Data":"1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358"}
Sep 16 05:04:13.617132 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:13.616966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc" event={"ID":"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7","Type":"ContainerStarted","Data":"82f4684c43d6c92d58f76a9c142bb4329fe70deb5395b99b6f5e0aa56c4594cd"}
Sep 16 05:04:13.617132 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:13.616999    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:13.618443 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:13.618417    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc" event={"ID":"c69c08a6-9ae5-4dad-9ee8-446bc54088c0","Type":"ContainerStarted","Data":"635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc"}
Sep 16 05:04:13.618567 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:13.618492    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:13.633549 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:13.633502    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc" podStartSLOduration=1.6334891329999999 podStartE2EDuration="1.633489133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:04:13.63158971 +0000 UTC m=+917.297519327" watchObservedRunningTime="2026-09-16 05:04:13.633489133 +0000 UTC m=+917.299418726"
Sep 16 05:04:13.646002 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:13.645955    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc" podStartSLOduration=2.146998262 podStartE2EDuration="2.645942508s" totalImagesPullingTime="498.944246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:04:13.64557514 +0000 UTC m=+917.311504754" watchObservedRunningTime="2026-09-16 05:04:13.645942508 +0000 UTC m=+917.311872100"
Sep 16 05:04:14.030516 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:14.030481    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:14.665790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:14.665759    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:19.629382 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:19.629353    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:20.889811 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:20.889780    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:04:24.626075 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:24.626049    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:33.342707 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:33.342674    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:35.070017 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:35.069983    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:35.543054 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:35.543024    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:36.241034 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:36.241003    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:37.047353 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:37.047322    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:37.700793 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:37.700763    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:38.321959 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:38.321926    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:38.942983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:38.942956    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:39.538318 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:39.538282    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:40.176516 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:40.176477    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:40.935177 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:40.935145    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"]
Sep 16 05:04:40.935439 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:40.935356    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc" podUID="c69c08a6-9ae5-4dad-9ee8-446bc54088c0" containerName="maas-api" containerID="cri-o://635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc" gracePeriod=30
Sep 16 05:04:40.957411 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:40.957389    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:40.985835 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:40.985807    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"]
Sep 16 05:04:40.986034 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:40.986011    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc" podUID="8760f52c-c6d5-4c2f-84ed-35d04b0f35f7" containerName="payload-pre-processing" containerID="cri-o://1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358" gracePeriod=30
Sep 16 05:04:41.056394 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.056368    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:41.150868 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.150834    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:41.216110 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.216034    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:41.233805 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.233781    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:41.266202 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.265220    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-maas-api-tls\") pod \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") "
Sep 16 05:04:41.266202 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.265379    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-openshift-service-ca\") pod \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") "
Sep 16 05:04:41.266202 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.265454    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-metrics-certs\") pod \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") "
Sep 16 05:04:41.266202 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.265498    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dwpn\" (UniqueName: \"kubernetes.io/projected/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-kube-api-access-6dwpn\") pod \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\" (UID: \"c69c08a6-9ae5-4dad-9ee8-446bc54088c0\") "
Sep 16 05:04:41.266918 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.266886    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c69c08a6-9ae5-4dad-9ee8-446bc54088c0" (UID: "c69c08a6-9ae5-4dad-9ee8-446bc54088c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:04:41.269120 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.269097    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-kube-api-access-6dwpn" (OuterVolumeSpecName: "kube-api-access-6dwpn") pod "c69c08a6-9ae5-4dad-9ee8-446bc54088c0" (UID: "c69c08a6-9ae5-4dad-9ee8-446bc54088c0"). InnerVolumeSpecName "kube-api-access-6dwpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:04:41.269501 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.269480    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c69c08a6-9ae5-4dad-9ee8-446bc54088c0" (UID: "c69c08a6-9ae5-4dad-9ee8-446bc54088c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:04:41.269989 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.269962    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c69c08a6-9ae5-4dad-9ee8-446bc54088c0" (UID: "c69c08a6-9ae5-4dad-9ee8-446bc54088c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:04:41.272096 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.272075    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:41.366821 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.366798    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwt5f\" (UniqueName: \"kubernetes.io/projected/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-kube-api-access-lwt5f\") pod \"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7\" (UID: \"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7\") "
Sep 16 05:04:41.366935 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.366826    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-plugins-config-volume\") pod \"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7\" (UID: \"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7\") "
Sep 16 05:04:41.367133 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.367113    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:41.367227 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.367137    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:41.367227 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.367152    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwpn\" (UniqueName: \"kubernetes.io/projected/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-kube-api-access-6dwpn\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:41.367227 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.367164    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c08a6-9ae5-4dad-9ee8-446bc54088c0-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:41.367352 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.367249    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8760f52c-c6d5-4c2f-84ed-35d04b0f35f7" (UID: "8760f52c-c6d5-4c2f-84ed-35d04b0f35f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:04:41.368797 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.368779    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-kube-api-access-lwt5f" (OuterVolumeSpecName: "kube-api-access-lwt5f") pod "8760f52c-c6d5-4c2f-84ed-35d04b0f35f7" (UID: "8760f52c-c6d5-4c2f-84ed-35d04b0f35f7"). InnerVolumeSpecName "kube-api-access-lwt5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:04:41.468325 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.468268    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwt5f\" (UniqueName: \"kubernetes.io/projected/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-kube-api-access-lwt5f\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:41.468325 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.468289    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:04:41.538370 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.538350    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:41.760015 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.759948    2573 generic.go:356] "Generic (PLEG): container finished" podID="8760f52c-c6d5-4c2f-84ed-35d04b0f35f7" containerID="1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358" exitCode=0
Sep 16 05:04:41.760015 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.760005    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"
Sep 16 05:04:41.760194 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.760000    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc" event={"ID":"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7","Type":"ContainerDied","Data":"1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358"}
Sep 16 05:04:41.760194 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.760156    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc" event={"ID":"8760f52c-c6d5-4c2f-84ed-35d04b0f35f7","Type":"ContainerDied","Data":"82f4684c43d6c92d58f76a9c142bb4329fe70deb5395b99b6f5e0aa56c4594cd"}
Sep 16 05:04:41.760194 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.760183    2573 scope.go:117] "RemoveContainer" containerID="1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358"
Sep 16 05:04:41.761535 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.761512    2573 generic.go:356] "Generic (PLEG): container finished" podID="c69c08a6-9ae5-4dad-9ee8-446bc54088c0" containerID="635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc" exitCode=0
Sep 16 05:04:41.761668 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.761576    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"
Sep 16 05:04:41.761668 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.761620    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc" event={"ID":"c69c08a6-9ae5-4dad-9ee8-446bc54088c0","Type":"ContainerDied","Data":"635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc"}
Sep 16 05:04:41.761668 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.761662    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc" event={"ID":"c69c08a6-9ae5-4dad-9ee8-446bc54088c0","Type":"ContainerDied","Data":"3ecd691f68b630df682ff64e8896d96489d25e09f5ab496845662d6b3fefca40"}
Sep 16 05:04:41.768404 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.768386    2573 scope.go:117] "RemoveContainer" containerID="1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358"
Sep 16 05:04:41.768688 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:04:41.768670    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358\": container with ID starting with 1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358 not found: ID does not exist" containerID="1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358"
Sep 16 05:04:41.768756 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.768695    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358"} err="failed to get container status \"1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358\": rpc error: code = NotFound desc = could not find container \"1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358\": container with ID starting with 1525bb2b1c599d07e77b132ba1d96bb3d019ebc915af5a14a003c3acd7b76358 not found: ID does not exist"
Sep 16 05:04:41.768756 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.768712    2573 scope.go:117] "RemoveContainer" containerID="635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc"
Sep 16 05:04:41.776873 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.776857    2573 scope.go:117] "RemoveContainer" containerID="635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc"
Sep 16 05:04:41.777150 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:04:41.777132    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc\": container with ID starting with 635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc not found: ID does not exist" containerID="635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc"
Sep 16 05:04:41.777199 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.777159    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc"} err="failed to get container status \"635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc\": rpc error: code = NotFound desc = could not find container \"635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc\": container with ID starting with 635d3f15dd178ea5ce0317804f45dd121b6375fd5e7f5ee8251557d197fa4bdc not found: ID does not exist"
Sep 16 05:04:41.784395 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.784369    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"]
Sep 16 05:04:41.788130 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.788110    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25f3e8f3-79fb49d64d-qdcpc"]
Sep 16 05:04:41.797659 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.797640    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"]
Sep 16 05:04:41.802156 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:41.802139    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25f3e8f3-5c9f74798-xsvpc"]
Sep 16 05:04:42.174547 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:42.174519    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:42.270677 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:42.270643    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:42.655249 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:42.655224    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:42.785152 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:42.785123    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8760f52c-c6d5-4c2f-84ed-35d04b0f35f7" path="/var/lib/kubelet/pods/8760f52c-c6d5-4c2f-84ed-35d04b0f35f7/volumes"
Sep 16 05:04:42.785498 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:42.785485    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c69c08a6-9ae5-4dad-9ee8-446bc54088c0" path="/var/lib/kubelet/pods/c69c08a6-9ae5-4dad-9ee8-446bc54088c0/volumes"
Sep 16 05:04:42.955468 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:42.955382    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:43.049738 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:43.049713    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:43.149996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:43.149969    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:43.733551 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:43.733527    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:44.331936 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:44.331910    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:45.024430 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:45.024384    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:45.818995 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:45.818957    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:45.918321 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:45.918276    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:46.020700 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:46.020670    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:46.510085 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:46.510043    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:47.233885 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:47.233855    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:47.381746 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:47.381715    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:04:48.007888 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:48.007856    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:48.702569 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:48.702535    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:49.410938 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:49.410903    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:50.004839 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.004801    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"]
Sep 16 05:04:50.005222 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.005204    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8760f52c-c6d5-4c2f-84ed-35d04b0f35f7" containerName="payload-pre-processing"
Sep 16 05:04:50.005312 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.005224    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8760f52c-c6d5-4c2f-84ed-35d04b0f35f7" containerName="payload-pre-processing"
Sep 16 05:04:50.005312 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.005247    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69c08a6-9ae5-4dad-9ee8-446bc54088c0" containerName="maas-api"
Sep 16 05:04:50.005312 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.005256    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69c08a6-9ae5-4dad-9ee8-446bc54088c0" containerName="maas-api"
Sep 16 05:04:50.005466 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.005336    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8760f52c-c6d5-4c2f-84ed-35d04b0f35f7" containerName="payload-pre-processing"
Sep 16 05:04:50.005466 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.005349    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69c08a6-9ae5-4dad-9ee8-446bc54088c0" containerName="maas-api"
Sep 16 05:04:50.009968 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.009949    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.011753 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.011730    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c9c153e4-serving-cert\""
Sep 16 05:04:50.017008 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.016988    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"]
Sep 16 05:04:50.034512 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.034490    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:50.091883 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.091858    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:50.137648 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.137619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b38b933-5cbd-4446-a9e6-763e5c750439-openshift-service-ca\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.137772 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.137666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-metrics-certs\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.137823 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.137763    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-maas-api-tls\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.137823 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.137794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfnk\" (UniqueName: \"kubernetes.io/projected/2b38b933-5cbd-4446-a9e6-763e5c750439-kube-api-access-vlfnk\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.238845 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.238819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b38b933-5cbd-4446-a9e6-763e5c750439-openshift-service-ca\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.238973 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.238852    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-metrics-certs\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.238973 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.238889    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-maas-api-tls\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.238973 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.238905    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfnk\" (UniqueName: \"kubernetes.io/projected/2b38b933-5cbd-4446-a9e6-763e5c750439-kube-api-access-vlfnk\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.239462 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.239441    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b38b933-5cbd-4446-a9e6-763e5c750439-openshift-service-ca\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.241438 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.241417    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-metrics-certs\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.241535 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.241450    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-maas-api-tls\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.246302 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.246282    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfnk\" (UniqueName: \"kubernetes.io/projected/2b38b933-5cbd-4446-a9e6-763e5c750439-kube-api-access-vlfnk\") pod \"maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.270678 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.270628    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:50.322159 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.322133    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:50.461238 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.461212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:50.561140 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.561069    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:04:50.592249 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.592221    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"]
Sep 16 05:04:50.595850 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:04:50.595821    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b38b933_5cbd_4446_a9e6_763e5c750439.slice/crio-2ea362baa82fea8d9a603bbae6a2d1c66b18c15bc21387624d913a08912936cb WatchSource:0}: Error finding container 2ea362baa82fea8d9a603bbae6a2d1c66b18c15bc21387624d913a08912936cb: Status 404 returned error can't find the container with id 2ea362baa82fea8d9a603bbae6a2d1c66b18c15bc21387624d913a08912936cb
Sep 16 05:04:50.601706 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.601682    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"]
Sep 16 05:04:50.606261 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.606245    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:04:50.608113 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.608095    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c9c153e4-dockercfg-fgwsh\""
Sep 16 05:04:50.608172 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.608095    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c9c153e4\""
Sep 16 05:04:50.613068 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.613048    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"]
Sep 16 05:04:50.642703 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.642682    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afc63260-5ca8-4074-95c9-811c0d13a1a4-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng\" (UID: \"afc63260-5ca8-4074-95c9-811c0d13a1a4\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:04:50.642806 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.642717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46ncw\" (UniqueName: \"kubernetes.io/projected/afc63260-5ca8-4074-95c9-811c0d13a1a4-kube-api-access-46ncw\") pod \"payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng\" (UID: \"afc63260-5ca8-4074-95c9-811c0d13a1a4\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:04:50.743933 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.743905    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afc63260-5ca8-4074-95c9-811c0d13a1a4-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng\" (UID: \"afc63260-5ca8-4074-95c9-811c0d13a1a4\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:04:50.744078 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.743944    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46ncw\" (UniqueName: \"kubernetes.io/projected/afc63260-5ca8-4074-95c9-811c0d13a1a4-kube-api-access-46ncw\") pod \"payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng\" (UID: \"afc63260-5ca8-4074-95c9-811c0d13a1a4\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:04:50.744616 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.744572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afc63260-5ca8-4074-95c9-811c0d13a1a4-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng\" (UID: \"afc63260-5ca8-4074-95c9-811c0d13a1a4\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:04:50.752805 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.752787    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46ncw\" (UniqueName: \"kubernetes.io/projected/afc63260-5ca8-4074-95c9-811c0d13a1a4-kube-api-access-46ncw\") pod \"payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng\" (UID: \"afc63260-5ca8-4074-95c9-811c0d13a1a4\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:04:50.805556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.805531    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz" event={"ID":"2b38b933-5cbd-4446-a9e6-763e5c750439","Type":"ContainerStarted","Data":"2ea362baa82fea8d9a603bbae6a2d1c66b18c15bc21387624d913a08912936cb"}
Sep 16 05:04:50.919090 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:50.919063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:04:51.211154 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:51.211128    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"]
Sep 16 05:04:51.213402 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:04:51.213377    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc63260_5ca8_4074_95c9_811c0d13a1a4.slice/crio-21a3c351de1bfb7e6061337ad086b8d988155b601a64ee366be4fa97d59aba2b WatchSource:0}: Error finding container 21a3c351de1bfb7e6061337ad086b8d988155b601a64ee366be4fa97d59aba2b: Status 404 returned error can't find the container with id 21a3c351de1bfb7e6061337ad086b8d988155b601a64ee366be4fa97d59aba2b
Sep 16 05:04:51.811873 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:51.811835    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng" event={"ID":"afc63260-5ca8-4074-95c9-811c0d13a1a4","Type":"ContainerStarted","Data":"a02e7c199fd62db7f974f6ae33d295eccbf4ac449b23e5a85d5aff396c0cca61"}
Sep 16 05:04:51.812330 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:51.811896    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:04:51.812330 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:51.811916    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng" event={"ID":"afc63260-5ca8-4074-95c9-811c0d13a1a4","Type":"ContainerStarted","Data":"21a3c351de1bfb7e6061337ad086b8d988155b601a64ee366be4fa97d59aba2b"}
Sep 16 05:04:51.813535 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:51.813508    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz" event={"ID":"2b38b933-5cbd-4446-a9e6-763e5c750439","Type":"ContainerStarted","Data":"eb39ddf9f218ebd0dc4252cc58da9cfdfd6de1c08c84f3ab2b1757e8098e0bd8"}
Sep 16 05:04:51.813671 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:51.813564    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:04:51.830512 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:51.830458    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng" podStartSLOduration=1.8304389460000001 podStartE2EDuration="1.830438946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:04:51.827663459 +0000 UTC m=+955.493593063" watchObservedRunningTime="2026-09-16 05:04:51.830438946 +0000 UTC m=+955.496368540"
Sep 16 05:04:51.844776 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:51.844740    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz" podStartSLOduration=2.427858676 podStartE2EDuration="2.844726992s" totalImagesPullingTime="416.868316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:04:51.842656904 +0000 UTC m=+955.508586524" watchObservedRunningTime="2026-09-16 05:04:51.844726992 +0000 UTC m=+955.510656584"
Sep 16 05:04:57.822807 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:04:57.822775    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:05:02.818882 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:02.818850    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:05:03.986759 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:03.986722    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:05:05.076015 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:05.075974    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:11.909942 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:11.909909    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:12.733969 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:12.733942    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:13.370058 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:13.370020    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:14.133114 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:14.133079    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:14.880591 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:14.880552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:15.624502 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:15.624467    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:16.365729 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:16.365698    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:17.028933 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.028907    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:17.353642 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.353550    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"]
Sep 16 05:05:17.358478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.358455    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.361186 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.361166    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee7a60\"/\"simulator-w0-ee7a60-kserve-self-signed-certs\""
Sep 16 05:05:17.361186 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.361177    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee7a60\"/\"kube-root-ca.crt\""
Sep 16 05:05:17.361346 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.361167    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee7a60\"/\"openshift-service-ca.crt\""
Sep 16 05:05:17.361346 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.361171    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee7a60\"/\"default-dockercfg-jzbds\""
Sep 16 05:05:17.368056 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.368036    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"]
Sep 16 05:05:17.378896 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.378877    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:17.443824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.443803    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2rv2\" (UniqueName: \"kubernetes.io/projected/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-kube-api-access-x2rv2\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.443926 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.443839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-dshm\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.443926 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.443857    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-home\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.443926 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.443874    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tls-certs\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.444078 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.443938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tmp-dir\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.444078 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.444018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-model-cache\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.545143 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.545118    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-dshm\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.545267 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.545149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-home\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.545267 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.545176    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tls-certs\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.545267 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.545199    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tmp-dir\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.545267 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.545260    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-model-cache\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.545486 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.545331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rv2\" (UniqueName: \"kubernetes.io/projected/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-kube-api-access-x2rv2\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.545641 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.545588    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-home\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.545641 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.545637    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tmp-dir\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.545791 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.545681    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-model-cache\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.547366 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.547341    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-dshm\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.547648 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.547631    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tls-certs\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.554118 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.554095    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rv2\" (UniqueName: \"kubernetes.io/projected/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-kube-api-access-x2rv2\") pod \"simulator-w0-ee7a60-kserve-97bbc548f-w4k2b\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") " pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.668165 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.668141    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:17.930988 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.930909    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:17.939531 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:17.939474    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"]
Sep 16 05:05:17.941636 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:05:17.941610    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7c8b0ae_4ef6_4f77_abd8_0ffb114e6fde.slice/crio-5b43014ba6ae67217b8bf022156a569dd8e423f078394f239f65fa4a5aa157c2 WatchSource:0}: Error finding container 5b43014ba6ae67217b8bf022156a569dd8e423f078394f239f65fa4a5aa157c2: Status 404 returned error can't find the container with id 5b43014ba6ae67217b8bf022156a569dd8e423f078394f239f65fa4a5aa157c2
Sep 16 05:05:18.561788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:18.561763    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:18.940145 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:18.940106    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b" event={"ID":"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde","Type":"ContainerStarted","Data":"879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c"}
Sep 16 05:05:18.940346 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:18.940159    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b" event={"ID":"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde","Type":"ContainerStarted","Data":"5b43014ba6ae67217b8bf022156a569dd8e423f078394f239f65fa4a5aa157c2"}
Sep 16 05:05:18.955219 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:18.955173    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b" podStartSLOduration=1.955158197 podStartE2EDuration="1.955158197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:05:18.953790252 +0000 UTC m=+982.619719846" watchObservedRunningTime="2026-09-16 05:05:18.955158197 +0000 UTC m=+982.621087835"
Sep 16 05:05:19.183786 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:19.183757    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:19.945266 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:19.945193    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:20.578156 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:20.578123    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:21.777550 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:21.777519    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:05:24.340414 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:24.340386    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:26.989111 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:26.989085    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:27.180774 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.180743    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:27.669052 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.669024    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:27.669297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.669280    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:27.681504 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.681480    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:27.700573 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.700545    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"]
Sep 16 05:05:27.700780 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.700757    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz" podUID="2b38b933-5cbd-4446-a9e6-763e5c750439" containerName="maas-api" containerID="cri-o://eb39ddf9f218ebd0dc4252cc58da9cfdfd6de1c08c84f3ab2b1757e8098e0bd8" gracePeriod=30
Sep 16 05:05:27.757491 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.757465    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"]
Sep 16 05:05:27.757718 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.757694    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng" podUID="afc63260-5ca8-4074-95c9-811c0d13a1a4" containerName="payload-processing" containerID="cri-o://a02e7c199fd62db7f974f6ae33d295eccbf4ac449b23e5a85d5aff396c0cca61" gracePeriod=30
Sep 16 05:05:27.760281 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.760251    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:27.818085 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.818050    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz" podUID="2b38b933-5cbd-4446-a9e6-763e5c750439" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.42:8443/health\": dial tcp 10.133.0.42:8443: connect: connection refused"
Sep 16 05:05:27.848547 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.848523    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:27.908233 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.908188    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:27.986788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.986760    2573 generic.go:356] "Generic (PLEG): container finished" podID="afc63260-5ca8-4074-95c9-811c0d13a1a4" containerID="a02e7c199fd62db7f974f6ae33d295eccbf4ac449b23e5a85d5aff396c0cca61" exitCode=0
Sep 16 05:05:27.986927 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.986832    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng" event={"ID":"afc63260-5ca8-4074-95c9-811c0d13a1a4","Type":"ContainerDied","Data":"a02e7c199fd62db7f974f6ae33d295eccbf4ac449b23e5a85d5aff396c0cca61"}
Sep 16 05:05:27.988346 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.988320    2573 generic.go:356] "Generic (PLEG): container finished" podID="2b38b933-5cbd-4446-a9e6-763e5c750439" containerID="eb39ddf9f218ebd0dc4252cc58da9cfdfd6de1c08c84f3ab2b1757e8098e0bd8" exitCode=0
Sep 16 05:05:27.988468 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:27.988386    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz" event={"ID":"2b38b933-5cbd-4446-a9e6-763e5c750439","Type":"ContainerDied","Data":"eb39ddf9f218ebd0dc4252cc58da9cfdfd6de1c08c84f3ab2b1757e8098e0bd8"}
Sep 16 05:05:28.003983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.003963    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:05:28.013519 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.013499    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:05:28.016297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.016254    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:28.047556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.047532    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:05:28.091897 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.091869    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:28.130624 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.130583    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b38b933-5cbd-4446-a9e6-763e5c750439-openshift-service-ca\") pod \"2b38b933-5cbd-4446-a9e6-763e5c750439\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") "
Sep 16 05:05:28.130771 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.130647    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-maas-api-tls\") pod \"2b38b933-5cbd-4446-a9e6-763e5c750439\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") "
Sep 16 05:05:28.130771 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.130667    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlfnk\" (UniqueName: \"kubernetes.io/projected/2b38b933-5cbd-4446-a9e6-763e5c750439-kube-api-access-vlfnk\") pod \"2b38b933-5cbd-4446-a9e6-763e5c750439\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") "
Sep 16 05:05:28.130771 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.130726    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-metrics-certs\") pod \"2b38b933-5cbd-4446-a9e6-763e5c750439\" (UID: \"2b38b933-5cbd-4446-a9e6-763e5c750439\") "
Sep 16 05:05:28.130962 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.130936    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b38b933-5cbd-4446-a9e6-763e5c750439-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b38b933-5cbd-4446-a9e6-763e5c750439" (UID: "2b38b933-5cbd-4446-a9e6-763e5c750439"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:05:28.132764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.132735    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b38b933-5cbd-4446-a9e6-763e5c750439" (UID: "2b38b933-5cbd-4446-a9e6-763e5c750439"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:05:28.132863 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.132778    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2b38b933-5cbd-4446-a9e6-763e5c750439" (UID: "2b38b933-5cbd-4446-a9e6-763e5c750439"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:05:28.133012 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.132998    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b38b933-5cbd-4446-a9e6-763e5c750439-kube-api-access-vlfnk" (OuterVolumeSpecName: "kube-api-access-vlfnk") pod "2b38b933-5cbd-4446-a9e6-763e5c750439" (UID: "2b38b933-5cbd-4446-a9e6-763e5c750439"). InnerVolumeSpecName "kube-api-access-vlfnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:05:28.231662 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.231570    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afc63260-5ca8-4074-95c9-811c0d13a1a4-plugins-config-volume\") pod \"afc63260-5ca8-4074-95c9-811c0d13a1a4\" (UID: \"afc63260-5ca8-4074-95c9-811c0d13a1a4\") "
Sep 16 05:05:28.231662 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.231658    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46ncw\" (UniqueName: \"kubernetes.io/projected/afc63260-5ca8-4074-95c9-811c0d13a1a4-kube-api-access-46ncw\") pod \"afc63260-5ca8-4074-95c9-811c0d13a1a4\" (UID: \"afc63260-5ca8-4074-95c9-811c0d13a1a4\") "
Sep 16 05:05:28.231885 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.231865    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b38b933-5cbd-4446-a9e6-763e5c750439-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:05:28.231885 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.231883    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:05:28.231990 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.231897    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlfnk\" (UniqueName: \"kubernetes.io/projected/2b38b933-5cbd-4446-a9e6-763e5c750439-kube-api-access-vlfnk\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:05:28.231990 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.231911    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b38b933-5cbd-4446-a9e6-763e5c750439-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:05:28.231990 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.231940    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc63260-5ca8-4074-95c9-811c0d13a1a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "afc63260-5ca8-4074-95c9-811c0d13a1a4" (UID: "afc63260-5ca8-4074-95c9-811c0d13a1a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:05:28.234093 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.234064    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc63260-5ca8-4074-95c9-811c0d13a1a4-kube-api-access-46ncw" (OuterVolumeSpecName: "kube-api-access-46ncw") pod "afc63260-5ca8-4074-95c9-811c0d13a1a4" (UID: "afc63260-5ca8-4074-95c9-811c0d13a1a4"). InnerVolumeSpecName "kube-api-access-46ncw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:05:28.332763 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.332739    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afc63260-5ca8-4074-95c9-811c0d13a1a4-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:05:28.332763 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.332760    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46ncw\" (UniqueName: \"kubernetes.io/projected/afc63260-5ca8-4074-95c9-811c0d13a1a4-kube-api-access-46ncw\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:05:28.335142 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.335118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:28.709819 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.709790    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:28.806240 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.806218    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:28.887140 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.887117    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:28.994571 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.994515    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"
Sep 16 05:05:28.994571 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.994524    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz" event={"ID":"2b38b933-5cbd-4446-a9e6-763e5c750439","Type":"ContainerDied","Data":"2ea362baa82fea8d9a603bbae6a2d1c66b18c15bc21387624d913a08912936cb"}
Sep 16 05:05:28.994762 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.994638    2573 scope.go:117] "RemoveContainer" containerID="eb39ddf9f218ebd0dc4252cc58da9cfdfd6de1c08c84f3ab2b1757e8098e0bd8"
Sep 16 05:05:28.995976 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.995905    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"
Sep 16 05:05:28.995976 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:28.995943    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng" event={"ID":"afc63260-5ca8-4074-95c9-811c0d13a1a4","Type":"ContainerDied","Data":"21a3c351de1bfb7e6061337ad086b8d988155b601a64ee366be4fa97d59aba2b"}
Sep 16 05:05:29.003374 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:29.003359    2573 scope.go:117] "RemoveContainer" containerID="a02e7c199fd62db7f974f6ae33d295eccbf4ac449b23e5a85d5aff396c0cca61"
Sep 16 05:05:29.011046 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:29.011029    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"]
Sep 16 05:05:29.013626 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:29.013588    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c153e4-d65b9555b-vg5nz"]
Sep 16 05:05:29.023355 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:29.023332    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"]
Sep 16 05:05:29.030491 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:29.030473    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c153e4-b9cb8b7df-xnbng"]
Sep 16 05:05:29.239252 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:29.239225    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:30.064652 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:30.064621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:30.323321 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:30.323235    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:30.786366 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:30.786333    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b38b933-5cbd-4446-a9e6-763e5c750439" path="/var/lib/kubelet/pods/2b38b933-5cbd-4446-a9e6-763e5c750439/volumes"
Sep 16 05:05:30.786711 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:30.786697    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afc63260-5ca8-4074-95c9-811c0d13a1a4" path="/var/lib/kubelet/pods/afc63260-5ca8-4074-95c9-811c0d13a1a4/volumes"
Sep 16 05:05:30.836690 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:30.836667    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:31.474293 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:31.474260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:32.203824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:32.203799    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:36.581204 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:36.581165    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:36.681821 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:36.681795    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:36.779200 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:36.779179    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:36.879986 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:36.879962    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:37.272443 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:37.272364    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:39.245029 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:39.244993    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:39.960132 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:39.960105    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:40.745374 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:40.745340    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:41.366298 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:41.366262    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:42.015484 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:42.015454    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:42.652144 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:42.652115    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:43.283580 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:43.283551    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:46.579843 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:46.579809    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:05:47.672357 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:47.672328    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:48.194127 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:48.194095    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:49.797922 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:49.797896    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:49.816557 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:49.816531    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"]
Sep 16 05:05:49.816888 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:49.816723    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7" podUID="a09b053c-30cc-4292-b4ad-a9064e672a66" containerName="payload-processing" containerID="cri-o://fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5" gracePeriod=30
Sep 16 05:05:49.900385 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:49.900356    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:49.973514 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:49.973483    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:50.034492 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.034464    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:50.072116 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.072094    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:05:50.092548 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.092527    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68jfn\" (UniqueName: \"kubernetes.io/projected/a09b053c-30cc-4292-b4ad-a9064e672a66-kube-api-access-68jfn\") pod \"a09b053c-30cc-4292-b4ad-a9064e672a66\" (UID: \"a09b053c-30cc-4292-b4ad-a9064e672a66\") "
Sep 16 05:05:50.092698 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.092557    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09b053c-30cc-4292-b4ad-a9064e672a66-plugins-config-volume\") pod \"a09b053c-30cc-4292-b4ad-a9064e672a66\" (UID: \"a09b053c-30cc-4292-b4ad-a9064e672a66\") "
Sep 16 05:05:50.093016 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.092990    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09b053c-30cc-4292-b4ad-a9064e672a66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a09b053c-30cc-4292-b4ad-a9064e672a66" (UID: "a09b053c-30cc-4292-b4ad-a9064e672a66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:05:50.094834 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.094802    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09b053c-30cc-4292-b4ad-a9064e672a66-kube-api-access-68jfn" (OuterVolumeSpecName: "kube-api-access-68jfn") pod "a09b053c-30cc-4292-b4ad-a9064e672a66" (UID: "a09b053c-30cc-4292-b4ad-a9064e672a66"). InnerVolumeSpecName "kube-api-access-68jfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:05:50.097373 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.097351    2573 generic.go:356] "Generic (PLEG): container finished" podID="a09b053c-30cc-4292-b4ad-a9064e672a66" containerID="fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5" exitCode=0
Sep 16 05:05:50.097469 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.097408    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"
Sep 16 05:05:50.097469 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.097438    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7" event={"ID":"a09b053c-30cc-4292-b4ad-a9064e672a66","Type":"ContainerDied","Data":"fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5"}
Sep 16 05:05:50.097556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.097505    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7" event={"ID":"a09b053c-30cc-4292-b4ad-a9064e672a66","Type":"ContainerDied","Data":"0abb981045c1f208467a4465077c645a110a6382c56ed5491a94aedb8be7fb9d"}
Sep 16 05:05:50.097556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.097537    2573 scope.go:117] "RemoveContainer" containerID="fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5"
Sep 16 05:05:50.105874 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.105856    2573 scope.go:117] "RemoveContainer" containerID="fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5"
Sep 16 05:05:50.106156 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:05:50.106140    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5\": container with ID starting with fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5 not found: ID does not exist" containerID="fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5"
Sep 16 05:05:50.106219 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.106167    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5"} err="failed to get container status \"fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5\": rpc error: code = NotFound desc = could not find container \"fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5\": container with ID starting with fa9617510caab59eef66d8d8e10c0933af226392bd91562c58cc1100fe680ec5 not found: ID does not exist"
Sep 16 05:05:50.120224 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.120202    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"]
Sep 16 05:05:50.124017 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.123998    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d05eb5-7f59d477cf-tjsv7"]
Sep 16 05:05:50.193973 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.193950    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68jfn\" (UniqueName: \"kubernetes.io/projected/a09b053c-30cc-4292-b4ad-a9064e672a66-kube-api-access-68jfn\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:05:50.193973 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.193971    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09b053c-30cc-4292-b4ad-a9064e672a66-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:05:50.328154 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.328099    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:50.427919 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.427898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:50.786199 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:50.786169    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09b053c-30cc-4292-b4ad-a9064e672a66" path="/var/lib/kubelet/pods/a09b053c-30cc-4292-b4ad-a9064e672a66/volumes"
Sep 16 05:05:51.069468 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:51.069403    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:51.697434 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:51.697399    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:52.339972 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:52.339943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:52.944848 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:52.944814    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:05:55.406168 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:05:55.406136    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:03.172424 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:03.172396    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:04.679708 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:04.679676    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:06:09.083487 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:09.083453    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:09.183058 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:09.183033    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:14.481675 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:14.481639    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:19.881530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:19.881492    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:06:20.049088 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.049060    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"]
Sep 16 05:06:20.049396 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.049383    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b38b933-5cbd-4446-a9e6-763e5c750439" containerName="maas-api"
Sep 16 05:06:20.049396 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.049396    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b38b933-5cbd-4446-a9e6-763e5c750439" containerName="maas-api"
Sep 16 05:06:20.049526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.049411    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09b053c-30cc-4292-b4ad-a9064e672a66" containerName="payload-processing"
Sep 16 05:06:20.049526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.049417    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09b053c-30cc-4292-b4ad-a9064e672a66" containerName="payload-processing"
Sep 16 05:06:20.049526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.049422    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afc63260-5ca8-4074-95c9-811c0d13a1a4" containerName="payload-processing"
Sep 16 05:06:20.049526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.049428    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="afc63260-5ca8-4074-95c9-811c0d13a1a4" containerName="payload-processing"
Sep 16 05:06:20.049526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.049483    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="afc63260-5ca8-4074-95c9-811c0d13a1a4" containerName="payload-processing"
Sep 16 05:06:20.049526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.049490    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b38b933-5cbd-4446-a9e6-763e5c750439" containerName="maas-api"
Sep 16 05:06:20.049526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.049497    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09b053c-30cc-4292-b4ad-a9064e672a66" containerName="payload-processing"
Sep 16 05:06:20.053835 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.053813    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.056082 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.056058    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee7a60\"/\"premium-w0-ee7a60-kserve-self-signed-certs\""
Sep 16 05:06:20.061587 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.061567    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"]
Sep 16 05:06:20.075259 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.075238    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:20.123838 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.123816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-dshm\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.123956 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.123848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tls-certs\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.123956 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.123879    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk6qz\" (UniqueName: \"kubernetes.io/projected/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-kube-api-access-vk6qz\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.124074 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.123973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-home\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.124074 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.124018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-model-cache\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.124074 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.124044    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tmp-dir\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.224440 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.224370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tls-certs\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.224440 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.224416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk6qz\" (UniqueName: \"kubernetes.io/projected/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-kube-api-access-vk6qz\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.224639 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.224557    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-home\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.224639 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.224632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-model-cache\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.224762 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.224659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tmp-dir\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.224824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.224762    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-dshm\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.225011 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.224986    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-home\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.225011 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.225002    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-model-cache\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.225219 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.225186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tmp-dir\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.226962 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.226939    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-dshm\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.227044 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.226998    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tls-certs\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.231895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.231872    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk6qz\" (UniqueName: \"kubernetes.io/projected/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-kube-api-access-vk6qz\") pod \"premium-w0-ee7a60-kserve-6df8cb448-t8d4v\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") " pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.364372 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.364343    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:20.640363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:20.640326    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"]
Sep 16 05:06:20.642609 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:06:20.642567    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a58a94b_fe7b_4a2b_8176_64d2e1a4eab5.slice/crio-55aba706ab35438cf309d061e06f6d7d60a5fe532dd603a2c06bfd7a0e26fa80 WatchSource:0}: Error finding container 55aba706ab35438cf309d061e06f6d7d60a5fe532dd603a2c06bfd7a0e26fa80: Status 404 returned error can't find the container with id 55aba706ab35438cf309d061e06f6d7d60a5fe532dd603a2c06bfd7a0e26fa80
Sep 16 05:06:21.249811 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:21.249776    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v" event={"ID":"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5","Type":"ContainerStarted","Data":"a55a19a50e2e4db2629396f39aac641aad848d5088a264772c1e1de465188935"}
Sep 16 05:06:21.250207 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:21.249824    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v" event={"ID":"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5","Type":"ContainerStarted","Data":"55aba706ab35438cf309d061e06f6d7d60a5fe532dd603a2c06bfd7a0e26fa80"}
Sep 16 05:06:21.266403 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:21.266353    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v" podStartSLOduration=1.26633863 podStartE2EDuration="1.26633863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:06:21.264669493 +0000 UTC m=+1044.930599105" watchObservedRunningTime="2026-09-16 05:06:21.26633863 +0000 UTC m=+1044.932268232"
Sep 16 05:06:24.892618 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:24.892562    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:30.365014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:30.364974    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:30.365014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:30.365023    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:30.377151 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:30.377124    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:30.929218 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:30.929191    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:30.998232 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:30.998155    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:31.091726 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:31.091698    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:31.310380 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:31.310317    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:06:32.321051 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:32.321018    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:32.627607 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:32.627577    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:33.033155 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:33.033072    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:33.849367 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:33.849338    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:33.910218 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:33.910185    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:34.031804 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.031772    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"]
Sep 16 05:06:34.035066 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.035045    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.037851 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.037819    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d1f291\"/\"test-model-d1f291-kserve-self-signed-certs\""
Sep 16 05:06:34.037851 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.037836    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d1f291\"/\"default-dockercfg-p77vt\""
Sep 16 05:06:34.037851 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.037846    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d1f291\"/\"openshift-service-ca.crt\""
Sep 16 05:06:34.038033 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.037850    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d1f291\"/\"kube-root-ca.crt\""
Sep 16 05:06:34.045857 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.045835    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"]
Sep 16 05:06:34.129834 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.129807    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-home\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.129834 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.129834    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-dshm\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.130037 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.129851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tmp-dir\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.130037 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.129955    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ld9\" (UniqueName: \"kubernetes.io/projected/cfa51a65-31c9-4e04-b2c2-27d3053240a2-kube-api-access-29ld9\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.130037 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.129989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-model-cache\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.130037 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.130018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tls-certs\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.158006 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.157985    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:34.230894 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.230862    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-home\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.230894 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.230890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-dshm\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.231079 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.230907    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tmp-dir\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.231079 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.230951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ld9\" (UniqueName: \"kubernetes.io/projected/cfa51a65-31c9-4e04-b2c2-27d3053240a2-kube-api-access-29ld9\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.231079 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.230982    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-model-cache\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.231079 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.231004    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tls-certs\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.231323 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.231300    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-home\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.231381 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.231353    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tmp-dir\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.231438 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.231381    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-model-cache\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.233065 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.233045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-dshm\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.233317 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.233300    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tls-certs\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.241900 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.241882    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ld9\" (UniqueName: \"kubernetes.io/projected/cfa51a65-31c9-4e04-b2c2-27d3053240a2-kube-api-access-29ld9\") pod \"test-model-d1f291-kserve-68f68f9cc9-xkjdc\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") " pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.344702 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.344672    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:34.613983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:34.613944    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"]
Sep 16 05:06:34.616075 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:06:34.616046    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfa51a65_31c9_4e04_b2c2_27d3053240a2.slice/crio-f494aabaa086c31156fb24e680affea919b00af6dece4bcd365d16b1d57772b4 WatchSource:0}: Error finding container f494aabaa086c31156fb24e680affea919b00af6dece4bcd365d16b1d57772b4: Status 404 returned error can't find the container with id f494aabaa086c31156fb24e680affea919b00af6dece4bcd365d16b1d57772b4
Sep 16 05:06:35.323504 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:35.323464    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc" event={"ID":"cfa51a65-31c9-4e04-b2c2-27d3053240a2","Type":"ContainerStarted","Data":"4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225"}
Sep 16 05:06:35.323504 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:35.323509    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc" event={"ID":"cfa51a65-31c9-4e04-b2c2-27d3053240a2","Type":"ContainerStarted","Data":"f494aabaa086c31156fb24e680affea919b00af6dece4bcd365d16b1d57772b4"}
Sep 16 05:06:35.344374 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:35.344319    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc" podStartSLOduration=1.3442927390000001 podStartE2EDuration="1.344292739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:06:35.343034697 +0000 UTC m=+1059.008964308" watchObservedRunningTime="2026-09-16 05:06:35.344292739 +0000 UTC m=+1059.010222333"
Sep 16 05:06:35.367672 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:35.367644    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:36.032731 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:36.032699    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:36.126913 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:36.126887    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:36.767983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:36.767951    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:37.690456 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:37.690427    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:37.981373 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:37.981305    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:06:38.300036 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:38.299966    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:38.910897 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:38.910861    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"]
Sep 16 05:06:38.914575 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:38.914556    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:38.916572 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:38.916555    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-672f57-serving-cert\""
Sep 16 05:06:38.926718 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:38.926695    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"]
Sep 16 05:06:38.944551 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:38.944526    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:39.000318 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.000289    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:39.071560 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.071532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.071684 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.071566    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-metrics-certs\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.071684 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.071591    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a395d48-943f-41f5-ad48-3924f9467750-openshift-service-ca\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.071764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.071681    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxm4q\" (UniqueName: \"kubernetes.io/projected/7a395d48-943f-41f5-ad48-3924f9467750-kube-api-access-xxm4q\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.172226 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.172160    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxm4q\" (UniqueName: \"kubernetes.io/projected/7a395d48-943f-41f5-ad48-3924f9467750-kube-api-access-xxm4q\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.172370 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.172257    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.172370 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.172293    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-metrics-certs\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.172370 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.172326    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a395d48-943f-41f5-ad48-3924f9467750-openshift-service-ca\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.172547 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:06:39.172388    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-serving-cert: secret "maas-api-e2e-explicit-672f57-serving-cert" not found
Sep 16 05:06:39.172547 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:06:39.172454    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls podName:7a395d48-943f-41f5-ad48-3924f9467750 nodeName:}" failed. No retries permitted until 2026-09-16 05:06:39.672435661 +0000 UTC m=+1063.338365247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls") pod "maas-api-e2e-explicit-672f57-777c589649-jghw7" (UID: "7a395d48-943f-41f5-ad48-3924f9467750") : secret "maas-api-e2e-explicit-672f57-serving-cert" not found
Sep 16 05:06:39.173012 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.172990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a395d48-943f-41f5-ad48-3924f9467750-openshift-service-ca\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.174728 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.174708    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-metrics-certs\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.180187 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.180164    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxm4q\" (UniqueName: \"kubernetes.io/projected/7a395d48-943f-41f5-ad48-3924f9467750-kube-api-access-xxm4q\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.606446 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.606420    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:39.671984 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.671959    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:39.676155 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.676128    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:39.676279 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:06:39.676263    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-serving-cert: secret "maas-api-e2e-explicit-672f57-serving-cert" not found
Sep 16 05:06:39.676330 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:06:39.676323    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls podName:7a395d48-943f-41f5-ad48-3924f9467750 nodeName:}" failed. No retries permitted until 2026-09-16 05:06:40.676304542 +0000 UTC m=+1064.342234127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls") pod "maas-api-e2e-explicit-672f57-777c589649-jghw7" (UID: "7a395d48-943f-41f5-ad48-3924f9467750") : secret "maas-api-e2e-explicit-672f57-serving-cert" not found
Sep 16 05:06:39.771788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:39.771751    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:40.684540 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:40.684511    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:40.686859 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:40.686833    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls\") pod \"maas-api-e2e-explicit-672f57-777c589649-jghw7\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:40.727853 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:40.727822    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:41.012812 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:41.012746    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"]
Sep 16 05:06:41.015505 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:06:41.015478    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a395d48_943f_41f5_ad48_3924f9467750.slice/crio-cafe4f3a9c242899cb9c272e3cc8d6cbcd92abf8f5a93e8461ebaa085354e151 WatchSource:0}: Error finding container cafe4f3a9c242899cb9c272e3cc8d6cbcd92abf8f5a93e8461ebaa085354e151: Status 404 returned error can't find the container with id cafe4f3a9c242899cb9c272e3cc8d6cbcd92abf8f5a93e8461ebaa085354e151
Sep 16 05:06:41.016871 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:41.016853    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 05:06:41.352270 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:41.352193    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7" event={"ID":"7a395d48-943f-41f5-ad48-3924f9467750","Type":"ContainerStarted","Data":"cafe4f3a9c242899cb9c272e3cc8d6cbcd92abf8f5a93e8461ebaa085354e151"}
Sep 16 05:06:41.618439 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:41.618415    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:42.358292 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:42.358257    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7" event={"ID":"7a395d48-943f-41f5-ad48-3924f9467750","Type":"ContainerStarted","Data":"1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe"}
Sep 16 05:06:42.358688 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:42.358417    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:42.364468 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:42.364447    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:42.378712 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:42.378671    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7" podStartSLOduration=3.778444518 podStartE2EDuration="4.378656702s" totalImagesPullingTime="600.212184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:06:42.377835779 +0000 UTC m=+1066.043765409" watchObservedRunningTime="2026-09-16 05:06:42.378656702 +0000 UTC m=+1066.044586294"
Sep 16 05:06:44.345592 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:44.345559    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:44.345934 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:44.345632    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:44.357904 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:44.357882    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:44.379313 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:44.379291    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:06:48.133168 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:48.133128    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:48.300345 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:48.300298    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:48.368939 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:48.368918    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:06:48.392650 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:48.392550    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:49.027997 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:49.027965    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:49.126682 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:49.126652    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:49.222396 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:49.222370    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:50.723574 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:50.723489    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:56.384161 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:56.384132    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:56.449028 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:56.449006    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:57.055449 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:57.055422    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:06:59.575271 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:06:59.575235    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:07:00.966662 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:00.966621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:01.066926 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:01.066901    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:01.822226 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:01.822192    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:02.587248 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:02.587210    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:03.331021 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:03.330992    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:03.580108 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:03.580076    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:03.687967 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:03.687943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:03.824916 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:03.824890    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:04.128754 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:04.128728    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:04.249779 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:04.249748    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:04.853883 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:04.853852    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:05.268753 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:05.268724    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:09.146777 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:09.146740    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:09.934212 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:09.934184    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:10.020070 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:10.020037    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:10.847191 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:10.847160    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:11.840496 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:11.840445    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:12.473118 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:12.473085    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:13.249797 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:13.249748    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:13.911049 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:13.911020    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"]
Sep 16 05:07:13.914626 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:13.914607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:13.916822 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:13.916799    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a28799ba-serving-cert\""
Sep 16 05:07:13.929497 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:13.929475    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"]
Sep 16 05:07:13.944945 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:13.944920    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8hm\" (UniqueName: \"kubernetes.io/projected/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-kube-api-access-mx8hm\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:13.945043 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:13.944962    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-openshift-service-ca\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:13.945043 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:13.944986    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-maas-api-tls\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:13.945124 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:13.945091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-metrics-certs\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:13.948093 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:13.948072    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:14.045908 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.045875    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8hm\" (UniqueName: \"kubernetes.io/projected/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-kube-api-access-mx8hm\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:14.046059 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.045942    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-openshift-service-ca\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:14.046059 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.045993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-maas-api-tls\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:14.046059 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.046052    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-metrics-certs\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:14.046199 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:07:14.046171    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-serving-cert: secret "maas-api-e2e-mt-a28799ba-serving-cert" not found
Sep 16 05:07:14.046298 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:07:14.046276    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-maas-api-tls podName:007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd nodeName:}" failed. No retries permitted until 2026-09-16 05:07:14.54625686 +0000 UTC m=+1098.212186448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-maas-api-tls") pod "maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn" (UID: "007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd") : secret "maas-api-e2e-mt-a28799ba-serving-cert" not found
Sep 16 05:07:14.046763 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.046741    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:14.046864 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.046743    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-openshift-service-ca\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:14.048463 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.048444    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-metrics-certs\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:14.054055 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.054033    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8hm\" (UniqueName: \"kubernetes.io/projected/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-kube-api-access-mx8hm\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:14.160117 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.160096    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:14.413488 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.413451    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:14.517210 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.517184    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:14.551815 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.551775    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-maas-api-tls\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:14.554013 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.553990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-maas-api-tls\") pod \"maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:14.595107 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.595083    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:14.825728 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:14.825642    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:15.100521 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:15.100499    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"]
Sep 16 05:07:15.102378 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:07:15.102351    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007943b1_eb9a_4bc1_be9e_c2b6c2ae17bd.slice/crio-6fe34c379947e32ca7f5edd0b611c6ce8b8343cde5369d6efc4b9c7523c5cfe8 WatchSource:0}: Error finding container 6fe34c379947e32ca7f5edd0b611c6ce8b8343cde5369d6efc4b9c7523c5cfe8: Status 404 returned error can't find the container with id 6fe34c379947e32ca7f5edd0b611c6ce8b8343cde5369d6efc4b9c7523c5cfe8
Sep 16 05:07:15.516242 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:15.516209    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn" event={"ID":"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd","Type":"ContainerStarted","Data":"6fe34c379947e32ca7f5edd0b611c6ce8b8343cde5369d6efc4b9c7523c5cfe8"}
Sep 16 05:07:16.523611 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:16.523552    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn" event={"ID":"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd","Type":"ContainerStarted","Data":"980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41"}
Sep 16 05:07:16.524084 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:16.523735    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:16.541085 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:16.540998    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn" podStartSLOduration=3.095050188 podStartE2EDuration="3.540980989s" totalImagesPullingTime="445.930801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:07:16.540336172 +0000 UTC m=+1100.206265785" watchObservedRunningTime="2026-09-16 05:07:16.540980989 +0000 UTC m=+1100.206910582"
Sep 16 05:07:18.122323 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:18.122284    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:22.534517 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:22.534470    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:24.091164 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:24.091134    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:07:37.103032 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:37.103002    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:38.775144 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:38.775100    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:38.875181 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:38.875153    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:39.441775 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:39.441744    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:39.902108 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:39.902080    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:40.001590 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:40.001560    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:40.264960 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:40.264887    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:40.515277 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:40.515204    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:40.613352 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:40.613327    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:40.924146 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:40.924119    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:41.157801 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:41.157776    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:42.200112 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:42.200085    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:46.706884 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:46.706851    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:46.806665 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:46.806640    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:47.835409 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:47.835379    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:48.528088 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:48.528060    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:48.686522 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:48.686478    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:49.227201 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:49.227134    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:49.919606 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:49.919562    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:50.541806 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:50.541769    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:51.246132 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:51.246099    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:51.559148 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:51.559071    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:52.039769 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:52.039740    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:52.932462 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:52.932426    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:53.564287 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:53.564253    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:54.269388 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:54.269363    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:55.008908 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:55.008877    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:55.838052 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:55.838017    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:56.442062 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:56.442029    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:56.484962 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:56.484936    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:07:57.239814 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:57.239788    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:57.764831 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:57.764801    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"]
Sep 16 05:07:57.765066 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:57.765037    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn" podUID="007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd" containerName="maas-api" containerID="cri-o://980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41" gracePeriod=30
Sep 16 05:07:57.798912 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:57.798886    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:57.869356 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:57.869253    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:57.933192 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:57.933108    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:57.995179 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:57.995151    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:58.009115 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.009093    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:58.100623 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.100543    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-maas-api-tls\") pod \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") "
Sep 16 05:07:58.100623 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.100580    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-metrics-certs\") pod \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") "
Sep 16 05:07:58.100809 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.100627    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-openshift-service-ca\") pod \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") "
Sep 16 05:07:58.101034 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.101014    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd" (UID: "007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:07:58.102498 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.102468    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd" (UID: "007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:07:58.102586 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.102546    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd" (UID: "007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:07:58.201495 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.201464    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx8hm\" (UniqueName: \"kubernetes.io/projected/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-kube-api-access-mx8hm\") pod \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\" (UID: \"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd\") "
Sep 16 05:07:58.201677 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.201664    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:07:58.201727 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.201679    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:07:58.201727 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.201687    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:07:58.203348 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.203325    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-kube-api-access-mx8hm" (OuterVolumeSpecName: "kube-api-access-mx8hm") pod "007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd" (UID: "007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd"). InnerVolumeSpecName "kube-api-access-mx8hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:07:58.302815 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.302790    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx8hm\" (UniqueName: \"kubernetes.io/projected/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd-kube-api-access-mx8hm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:07:58.326979 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.326952    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:58.523242 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.523208    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:58.732308 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.732270    2573 generic.go:356] "Generic (PLEG): container finished" podID="007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd" containerID="980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41" exitCode=0
Sep 16 05:07:58.732481 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.732341    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"
Sep 16 05:07:58.732481 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.732358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn" event={"ID":"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd","Type":"ContainerDied","Data":"980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41"}
Sep 16 05:07:58.732481 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.732435    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn" event={"ID":"007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd","Type":"ContainerDied","Data":"6fe34c379947e32ca7f5edd0b611c6ce8b8343cde5369d6efc4b9c7523c5cfe8"}
Sep 16 05:07:58.732660 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.732455    2573 scope.go:117] "RemoveContainer" containerID="980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41"
Sep 16 05:07:58.741844 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.741829    2573 scope.go:117] "RemoveContainer" containerID="980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41"
Sep 16 05:07:58.742114 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:07:58.742095    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41\": container with ID starting with 980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41 not found: ID does not exist" containerID="980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41"
Sep 16 05:07:58.742162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.742122    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41"} err="failed to get container status \"980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41\": rpc error: code = NotFound desc = could not find container \"980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41\": container with ID starting with 980e4b71973fad0c9b15503481a55524404bf203cf529964006c85350b71da41 not found: ID does not exist"
Sep 16 05:07:58.751423 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.751404    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"]
Sep 16 05:07:58.757827 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.757809    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a28799ba-6844fbbfbb-jgxnn"]
Sep 16 05:07:58.786145 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:58.786127    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd" path="/var/lib/kubelet/pods/007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd/volumes"
Sep 16 05:07:59.151818 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:59.151791    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:07:59.948910 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:07:59.948882    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:00.593667 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:00.593636    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:01.439962 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:01.439935    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:02.134840 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:02.134813    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:02.727995 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:02.727960    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"]
Sep 16 05:08:02.728437 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:02.728149    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7" podUID="7a395d48-943f-41f5-ad48-3924f9467750" containerName="maas-api" containerID="cri-o://1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe" gracePeriod=30
Sep 16 05:08:02.780046 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:02.780018    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:02.881609 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:02.881564    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:02.967299 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:02.967275    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:08:02.972955 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:02.972934    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:03.032653 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.032520    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a395d48-943f-41f5-ad48-3924f9467750-openshift-service-ca\") pod \"7a395d48-943f-41f5-ad48-3924f9467750\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") "
Sep 16 05:08:03.032763 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.032656    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxm4q\" (UniqueName: \"kubernetes.io/projected/7a395d48-943f-41f5-ad48-3924f9467750-kube-api-access-xxm4q\") pod \"7a395d48-943f-41f5-ad48-3924f9467750\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") "
Sep 16 05:08:03.032763 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.032689    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls\") pod \"7a395d48-943f-41f5-ad48-3924f9467750\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") "
Sep 16 05:08:03.032763 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.032724    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-metrics-certs\") pod \"7a395d48-943f-41f5-ad48-3924f9467750\" (UID: \"7a395d48-943f-41f5-ad48-3924f9467750\") "
Sep 16 05:08:03.032914 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.032833    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a395d48-943f-41f5-ad48-3924f9467750-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a395d48-943f-41f5-ad48-3924f9467750" (UID: "7a395d48-943f-41f5-ad48-3924f9467750"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:08:03.032972 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.032915    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a395d48-943f-41f5-ad48-3924f9467750-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:08:03.034629 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.034588    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a395d48-943f-41f5-ad48-3924f9467750" (UID: "7a395d48-943f-41f5-ad48-3924f9467750"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:08:03.034728 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.034637    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a395d48-943f-41f5-ad48-3924f9467750" (UID: "7a395d48-943f-41f5-ad48-3924f9467750"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:08:03.034768 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.034738    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a395d48-943f-41f5-ad48-3924f9467750-kube-api-access-xxm4q" (OuterVolumeSpecName: "kube-api-access-xxm4q") pod "7a395d48-943f-41f5-ad48-3924f9467750" (UID: "7a395d48-943f-41f5-ad48-3924f9467750"). InnerVolumeSpecName "kube-api-access-xxm4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:08:03.038344 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.038295    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:03.134058 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.134030    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:08:03.134058 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.134056    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxm4q\" (UniqueName: \"kubernetes.io/projected/7a395d48-943f-41f5-ad48-3924f9467750-kube-api-access-xxm4q\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:08:03.134291 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.134074    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a395d48-943f-41f5-ad48-3924f9467750-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:08:03.321155 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.321080    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:03.401249 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.401220    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:03.758753 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.758723    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a395d48-943f-41f5-ad48-3924f9467750" containerID="1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe" exitCode=0
Sep 16 05:08:03.759133 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.758770    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"
Sep 16 05:08:03.759133 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.758802    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7" event={"ID":"7a395d48-943f-41f5-ad48-3924f9467750","Type":"ContainerDied","Data":"1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe"}
Sep 16 05:08:03.759133 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.758855    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7" event={"ID":"7a395d48-943f-41f5-ad48-3924f9467750","Type":"ContainerDied","Data":"cafe4f3a9c242899cb9c272e3cc8d6cbcd92abf8f5a93e8461ebaa085354e151"}
Sep 16 05:08:03.759133 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.758881    2573 scope.go:117] "RemoveContainer" containerID="1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe"
Sep 16 05:08:03.773967 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.773949    2573 scope.go:117] "RemoveContainer" containerID="1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe"
Sep 16 05:08:03.774209 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:08:03.774190    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe\": container with ID starting with 1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe not found: ID does not exist" containerID="1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe"
Sep 16 05:08:03.774256 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.774216    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe"} err="failed to get container status \"1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe\": rpc error: code = NotFound desc = could not find container \"1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe\": container with ID starting with 1476f9bfc7b8ba2e8f5e92c2fe291c73f82f0ff43e5646e4bda48a69117e79fe not found: ID does not exist"
Sep 16 05:08:03.781859 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.781836    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"]
Sep 16 05:08:03.785515 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:03.785496    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-672f57-777c589649-jghw7"]
Sep 16 05:08:04.076807 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:04.076742    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:04.711146 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:04.711119    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:04.785799 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:04.785770    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a395d48-943f-41f5-ad48-3924f9467750" path="/var/lib/kubelet/pods/7a395d48-943f-41f5-ad48-3924f9467750/volumes"
Sep 16 05:08:04.811112 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:04.811087    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:04.912489 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:04.912459    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:04.981018 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:04.980951    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:05.726347 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:05.726315    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:05.826109 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:05.826071    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:06.214430 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:06.214404    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:08.400857 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:08.400826    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:09.796296 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:09.796223    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:09.900152 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:09.900127    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:10.045731 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:10.045705    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:10.462389 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:10.462353    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:10.754388 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:10.754304    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:11.033524 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:11.033415    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:11.713106 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:11.713075    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:11.813785 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:11.813759    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:11.916886 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:11.916849    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:12.021372 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:12.021282    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:12.127302 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:12.127268    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:12.229835 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:12.229778    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:12.325005 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:12.324922    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:13.132464 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.132431    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"]
Sep 16 05:08:13.132997 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.132982    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd" containerName="maas-api"
Sep 16 05:08:13.133053 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.133000    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd" containerName="maas-api"
Sep 16 05:08:13.133053 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.133019    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a395d48-943f-41f5-ad48-3924f9467750" containerName="maas-api"
Sep 16 05:08:13.133053 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.133028    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a395d48-943f-41f5-ad48-3924f9467750" containerName="maas-api"
Sep 16 05:08:13.133158 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.133106    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="007943b1-eb9a-4bc1-be9e-c2b6c2ae17bd" containerName="maas-api"
Sep 16 05:08:13.133158 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.133118    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a395d48-943f-41f5-ad48-3924f9467750" containerName="maas-api"
Sep 16 05:08:13.138078 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.138051    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.140167 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.140140    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee7a60\"/\"distinct-w0-ee7a60-kserve-self-signed-certs\""
Sep 16 05:08:13.144050 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.144028    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"]
Sep 16 05:08:13.159903 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.159875    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:13.207792 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.207770    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tmp-dir\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.207895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.207801    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-dshm\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.207895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.207836    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tls-certs\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.208019 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.207929    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24w5s\" (UniqueName: \"kubernetes.io/projected/b6b6a982-d4fa-476d-87c5-45972dadf1c4-kube-api-access-24w5s\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.208019 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.207982    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-model-cache\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.208096 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.208033    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-home\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.309143 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.309114    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tls-certs\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.309242 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.309154    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24w5s\" (UniqueName: \"kubernetes.io/projected/b6b6a982-d4fa-476d-87c5-45972dadf1c4-kube-api-access-24w5s\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.309242 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.309191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-model-cache\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.309242 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.309215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-home\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.309369 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.309258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tmp-dir\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.309369 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.309292    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-dshm\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.309663 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.309643    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-model-cache\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.309752 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.309689    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-home\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.309752 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.309734    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tmp-dir\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.311488 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.311465    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-dshm\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.311670 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.311654    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tls-certs\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.316535 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.316516    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24w5s\" (UniqueName: \"kubernetes.io/projected/b6b6a982-d4fa-476d-87c5-45972dadf1c4-kube-api-access-24w5s\") pod \"distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") " pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.447563 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.447503    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:13.717318 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.717301    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"]
Sep 16 05:08:13.719555 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:08:13.719527    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6b6a982_d4fa_476d_87c5_45972dadf1c4.slice/crio-f9cbb9e55eceff8166bfea2416b9935a9d448399e6b942393db2f54b70add788 WatchSource:0}: Error finding container f9cbb9e55eceff8166bfea2416b9935a9d448399e6b942393db2f54b70add788: Status 404 returned error can't find the container with id f9cbb9e55eceff8166bfea2416b9935a9d448399e6b942393db2f54b70add788
Sep 16 05:08:13.807019 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.806989    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl" event={"ID":"b6b6a982-d4fa-476d-87c5-45972dadf1c4","Type":"ContainerStarted","Data":"d273b49f24feb546f721df546dc6ba06a6920ea9fec87283bb3f0dd0114ceb1c"}
Sep 16 05:08:13.807117 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.807035    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl" event={"ID":"b6b6a982-d4fa-476d-87c5-45972dadf1c4","Type":"ContainerStarted","Data":"f9cbb9e55eceff8166bfea2416b9935a9d448399e6b942393db2f54b70add788"}
Sep 16 05:08:13.823090 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:13.823035    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl" podStartSLOduration=0.823015882 podStartE2EDuration="823.015882ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:08:13.821779117 +0000 UTC m=+1157.487708750" watchObservedRunningTime="2026-09-16 05:08:13.823015882 +0000 UTC m=+1157.488945476"
Sep 16 05:08:17.531836 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:17.531782    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:17.783990 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:17.783918    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:17.884264 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:17.884235    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:18.662847 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:18.662814    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:18.928578 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:18.928501    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:19.039942 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:19.039912    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:19.251733 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:19.251663    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:21.120036 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:21.119999    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:21.669584 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:21.669552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:21.822359 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:21.822308    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:21.946990 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:21.946898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:22.733658 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:22.733630    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:23.448564 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:23.448488    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:23.448988 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:23.448646    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:23.464476 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:23.464409    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:23.553367 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:23.553342    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:23.866389 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:23.866318    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:08:24.228022 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:24.227985    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:24.852872 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:24.852831    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:24.907528 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:24.907500    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:25.079368 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.079341    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:25.289478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.289452    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:25.389058 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.389026    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:25.437661 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.437630    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"]
Sep 16 05:08:25.441054 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.441039    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:08:25.443090 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.443068    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ad06de-dockercfg-kbtq8\""
Sep 16 05:08:25.443199 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.443072    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ad06de\""
Sep 16 05:08:25.451514 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.451491    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"]
Sep 16 05:08:25.506022 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.505987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjclz\" (UniqueName: \"kubernetes.io/projected/83c190d7-6080-402d-b848-e4ee934374ee-kube-api-access-jjclz\") pod \"payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v\" (UID: \"83c190d7-6080-402d-b848-e4ee934374ee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:08:25.506135 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.506061    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c190d7-6080-402d-b848-e4ee934374ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v\" (UID: \"83c190d7-6080-402d-b848-e4ee934374ee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:08:25.607283 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.607218    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjclz\" (UniqueName: \"kubernetes.io/projected/83c190d7-6080-402d-b848-e4ee934374ee-kube-api-access-jjclz\") pod \"payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v\" (UID: \"83c190d7-6080-402d-b848-e4ee934374ee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:08:25.607283 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.607255    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c190d7-6080-402d-b848-e4ee934374ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v\" (UID: \"83c190d7-6080-402d-b848-e4ee934374ee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:08:25.607917 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.607899    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c190d7-6080-402d-b848-e4ee934374ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v\" (UID: \"83c190d7-6080-402d-b848-e4ee934374ee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:08:25.614773 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.614754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjclz\" (UniqueName: \"kubernetes.io/projected/83c190d7-6080-402d-b848-e4ee934374ee-kube-api-access-jjclz\") pod \"payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v\" (UID: \"83c190d7-6080-402d-b848-e4ee934374ee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:08:25.751722 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:25.751700    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:08:26.023544 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:26.023522    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"]
Sep 16 05:08:26.025769 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:08:26.025741    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c190d7_6080_402d_b848_e4ee934374ee.slice/crio-2ad39f9fe7dbe154c730d3c4c459abcffc04e4a4b4d74a546d6601aca9a7115b WatchSource:0}: Error finding container 2ad39f9fe7dbe154c730d3c4c459abcffc04e4a4b4d74a546d6601aca9a7115b: Status 404 returned error can't find the container with id 2ad39f9fe7dbe154c730d3c4c459abcffc04e4a4b4d74a546d6601aca9a7115b
Sep 16 05:08:26.870771 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:26.870734    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v" event={"ID":"83c190d7-6080-402d-b848-e4ee934374ee","Type":"ContainerStarted","Data":"6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e"}
Sep 16 05:08:26.870967 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:26.870784    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v" event={"ID":"83c190d7-6080-402d-b848-e4ee934374ee","Type":"ContainerStarted","Data":"2ad39f9fe7dbe154c730d3c4c459abcffc04e4a4b4d74a546d6601aca9a7115b"}
Sep 16 05:08:26.870967 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:26.870864    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:08:26.887411 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:26.887352    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v" podStartSLOduration=1.8873347740000002 podStartE2EDuration="1.887334774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:08:26.886249535 +0000 UTC m=+1170.552179141" watchObservedRunningTime="2026-09-16 05:08:26.887334774 +0000 UTC m=+1170.553264370"
Sep 16 05:08:28.891179 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:28.891148    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:08:33.266812 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:33.266779    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:34.953434 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:34.953406    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:37.878929 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:37.878893    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:08:43.158514 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:43.158481    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:45.741589 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:45.741557    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:47.184418 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:47.184382    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:47.937884 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:47.937856    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:48.566327 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:48.566298    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:49.320526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:49.320494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:50.166162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:50.166118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:50.757145 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:50.757113    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:51.712122 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:51.712070    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:52.081160 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:52.081068    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:52.355715 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:52.355621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:52.593999 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:52.593956    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:54.369852 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:54.369821    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:54.996332 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:54.996297    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:55.082761 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:55.082734    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:55.486080 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:55.486050    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:56.474962 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:56.474922    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:56.787421 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:56.787342    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:08:56.787812 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:56.787667    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:08:57.275365 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:57.275337    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:58.095031 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:58.095000    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:58.787134 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:58.787104    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:08:58.863143 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:58.863109    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:59.502275 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.502243    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"]
Sep 16 05:08:59.506755 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.506735    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.508847 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.508828    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-2ee234-serving-cert\""
Sep 16 05:08:59.516904 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.516883    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"]
Sep 16 05:08:59.536422 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.536401    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:59.610328 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.610304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2047c935-7175-4fae-b9ad-fd8c92b71c09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.610531 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.610347    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.610531 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.610368    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpblz\" (UniqueName: \"kubernetes.io/projected/2047c935-7175-4fae-b9ad-fd8c92b71c09-kube-api-access-lpblz\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.610666 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.610521    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.637288 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.637260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:59.689608 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.689573    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:08:59.711873 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.711846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.711966 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.711890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2047c935-7175-4fae-b9ad-fd8c92b71c09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.711966 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.711947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.712048 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.711972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpblz\" (UniqueName: \"kubernetes.io/projected/2047c935-7175-4fae-b9ad-fd8c92b71c09-kube-api-access-lpblz\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.712644 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.712611    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2047c935-7175-4fae-b9ad-fd8c92b71c09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.714447 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.714422    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.714528 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.714467    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.721946 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.721929    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpblz\" (UniqueName: \"kubernetes.io/projected/2047c935-7175-4fae-b9ad-fd8c92b71c09-kube-api-access-lpblz\") pod \"maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.818858 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.818796    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:08:59.986285 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:08:59.986248    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:00.086618 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:00.086516    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:00.100430 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:00.100406    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"]
Sep 16 05:09:00.103038 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:09:00.103016    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2047c935_7175_4fae_b9ad_fd8c92b71c09.slice/crio-2ab598594e430fce377b24fef0d0d8d4cb032c99217b310a5c425c612f99cade WatchSource:0}: Error finding container 2ab598594e430fce377b24fef0d0d8d4cb032c99217b310a5c425c612f99cade: Status 404 returned error can't find the container with id 2ab598594e430fce377b24fef0d0d8d4cb032c99217b310a5c425c612f99cade
Sep 16 05:09:01.032645 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:01.032576    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x" event={"ID":"2047c935-7175-4fae-b9ad-fd8c92b71c09","Type":"ContainerStarted","Data":"d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba"}
Sep 16 05:09:01.033008 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:01.032668    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x" event={"ID":"2047c935-7175-4fae-b9ad-fd8c92b71c09","Type":"ContainerStarted","Data":"2ab598594e430fce377b24fef0d0d8d4cb032c99217b310a5c425c612f99cade"}
Sep 16 05:09:01.052241 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:01.052189    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x" podStartSLOduration=1.546691764 podStartE2EDuration="2.052174236s" totalImagesPullingTime="505.482472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:01.049932404 +0000 UTC m=+1204.715861997" watchObservedRunningTime="2026-09-16 05:09:01.052174236 +0000 UTC m=+1204.718103828"
Sep 16 05:09:01.364814 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:01.364705    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:02.037154 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:02.037126    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:09:06.118821 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:06.118793    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:06.815095 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:06.815066    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:07.282737 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:07.282702    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:07.709729 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:07.709696    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:07.807493 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:07.807459    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:08.046560 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:08.046484    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:09:08.662155 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:08.662123    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:09.556953 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:09.556922    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:10.270906 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:10.270857    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:10.937074 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:10.937033    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:11.750443 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:11.750413    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:12.963377 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:12.963339    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:13.458207 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:13.458170    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:14.383388 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:14.383357    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:14.770691 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:14.770586    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:15.168377 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:15.168346    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:16.068208 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:16.068176    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:16.841339 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:16.841310    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:17.538209 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.538174    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"]
Sep 16 05:09:17.568590 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.568561    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"]
Sep 16 05:09:17.568811 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.568775    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h" podUID="3f5eaacb-14c9-42e2-beea-69e3afb989ee" containerName="maas-api" containerID="cri-o://69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7" gracePeriod=30
Sep 16 05:09:17.570621 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.570571    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:17.633225 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.633197    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"]
Sep 16 05:09:17.633441 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.633418    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5" podUID="b7c5e637-bb87-4301-9b70-65b8c3953dd0" containerName="payload-processing" containerID="cri-o://0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99" gracePeriod=30
Sep 16 05:09:17.666205 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.666172    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:17.737640 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.737538    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:17.829633 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.829573    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:17.875467 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.875441    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:09:17.906974 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.906952    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:09:17.911796 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.911772    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:17.977094 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.977065    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c5e637-bb87-4301-9b70-65b8c3953dd0-plugins-config-volume\") pod \"b7c5e637-bb87-4301-9b70-65b8c3953dd0\" (UID: \"b7c5e637-bb87-4301-9b70-65b8c3953dd0\") "
Sep 16 05:09:17.977206 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.977121    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-metrics-certs\") pod \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") "
Sep 16 05:09:17.977206 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.977193    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnmmf\" (UniqueName: \"kubernetes.io/projected/b7c5e637-bb87-4301-9b70-65b8c3953dd0-kube-api-access-rnmmf\") pod \"b7c5e637-bb87-4301-9b70-65b8c3953dd0\" (UID: \"b7c5e637-bb87-4301-9b70-65b8c3953dd0\") "
Sep 16 05:09:17.977335 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.977238    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9rbr\" (UniqueName: \"kubernetes.io/projected/3f5eaacb-14c9-42e2-beea-69e3afb989ee-kube-api-access-s9rbr\") pod \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") "
Sep 16 05:09:17.977392 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.977333    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-maas-api-tls\") pod \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") "
Sep 16 05:09:17.977449 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.977396    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5eaacb-14c9-42e2-beea-69e3afb989ee-openshift-service-ca\") pod \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\" (UID: \"3f5eaacb-14c9-42e2-beea-69e3afb989ee\") "
Sep 16 05:09:17.977504 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.977452    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c5e637-bb87-4301-9b70-65b8c3953dd0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7c5e637-bb87-4301-9b70-65b8c3953dd0" (UID: "b7c5e637-bb87-4301-9b70-65b8c3953dd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:09:17.977741 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.977719    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c5e637-bb87-4301-9b70-65b8c3953dd0-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:17.977897 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.977873    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5eaacb-14c9-42e2-beea-69e3afb989ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f5eaacb-14c9-42e2-beea-69e3afb989ee" (UID: "3f5eaacb-14c9-42e2-beea-69e3afb989ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:09:17.979889 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.979865    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f5eaacb-14c9-42e2-beea-69e3afb989ee" (UID: "3f5eaacb-14c9-42e2-beea-69e3afb989ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:09:17.979989 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.979951    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c5e637-bb87-4301-9b70-65b8c3953dd0-kube-api-access-rnmmf" (OuterVolumeSpecName: "kube-api-access-rnmmf") pod "b7c5e637-bb87-4301-9b70-65b8c3953dd0" (UID: "b7c5e637-bb87-4301-9b70-65b8c3953dd0"). InnerVolumeSpecName "kube-api-access-rnmmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:09:17.980052 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.980006    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f5eaacb-14c9-42e2-beea-69e3afb989ee" (UID: "3f5eaacb-14c9-42e2-beea-69e3afb989ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:09:17.980193 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:17.980167    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5eaacb-14c9-42e2-beea-69e3afb989ee-kube-api-access-s9rbr" (OuterVolumeSpecName: "kube-api-access-s9rbr") pod "3f5eaacb-14c9-42e2-beea-69e3afb989ee" (UID: "3f5eaacb-14c9-42e2-beea-69e3afb989ee"). InnerVolumeSpecName "kube-api-access-s9rbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:09:18.079049 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.078984    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9rbr\" (UniqueName: \"kubernetes.io/projected/3f5eaacb-14c9-42e2-beea-69e3afb989ee-kube-api-access-s9rbr\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:18.079049 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.079006    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:18.079049 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.079016    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5eaacb-14c9-42e2-beea-69e3afb989ee-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:18.079049 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.079025    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5eaacb-14c9-42e2-beea-69e3afb989ee-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:18.079049 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.079034    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmmf\" (UniqueName: \"kubernetes.io/projected/b7c5e637-bb87-4301-9b70-65b8c3953dd0-kube-api-access-rnmmf\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:18.116998 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.116966    2573 generic.go:356] "Generic (PLEG): container finished" podID="b7c5e637-bb87-4301-9b70-65b8c3953dd0" containerID="0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99" exitCode=0
Sep 16 05:09:18.117131 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.117012    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"
Sep 16 05:09:18.117131 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.117046    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5" event={"ID":"b7c5e637-bb87-4301-9b70-65b8c3953dd0","Type":"ContainerDied","Data":"0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99"}
Sep 16 05:09:18.117131 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.117101    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5" event={"ID":"b7c5e637-bb87-4301-9b70-65b8c3953dd0","Type":"ContainerDied","Data":"ae6d3e2156a3b90918487e22d68e5af5d69ead2adcb5f3120def913989393ba6"}
Sep 16 05:09:18.117131 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.117123    2573 scope.go:117] "RemoveContainer" containerID="0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99"
Sep 16 05:09:18.118696 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.118674    2573 generic.go:356] "Generic (PLEG): container finished" podID="3f5eaacb-14c9-42e2-beea-69e3afb989ee" containerID="69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7" exitCode=0
Sep 16 05:09:18.118791 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.118721    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"
Sep 16 05:09:18.118867 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.118727    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h" event={"ID":"3f5eaacb-14c9-42e2-beea-69e3afb989ee","Type":"ContainerDied","Data":"69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7"}
Sep 16 05:09:18.118867 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.118833    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h" event={"ID":"3f5eaacb-14c9-42e2-beea-69e3afb989ee","Type":"ContainerDied","Data":"869d1f963324a4b4da5ae21646fb5a90fe682d749c4513e04a3c8db8853df930"}
Sep 16 05:09:18.127459 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.127443    2573 scope.go:117] "RemoveContainer" containerID="0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99"
Sep 16 05:09:18.127778 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:09:18.127758    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99\": container with ID starting with 0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99 not found: ID does not exist" containerID="0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99"
Sep 16 05:09:18.127848 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.127787    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99"} err="failed to get container status \"0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99\": rpc error: code = NotFound desc = could not find container \"0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99\": container with ID starting with 0fdf767a9d4b2b6c1c46600c357cce1b736775c43b14a3e83d7b8f9576c7cf99 not found: ID does not exist"
Sep 16 05:09:18.127848 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.127802    2573 scope.go:117] "RemoveContainer" containerID="69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7"
Sep 16 05:09:18.136568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.136550    2573 scope.go:117] "RemoveContainer" containerID="69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7"
Sep 16 05:09:18.136879 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:09:18.136861    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7\": container with ID starting with 69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7 not found: ID does not exist" containerID="69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7"
Sep 16 05:09:18.136945 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.136885    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7"} err="failed to get container status \"69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7\": rpc error: code = NotFound desc = could not find container \"69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7\": container with ID starting with 69b9a5235ed1316769fd7c4192bbbcae37c837c3305a1bf1933218be74b9a2b7 not found: ID does not exist"
Sep 16 05:09:18.143801 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.143779    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"]
Sep 16 05:09:18.148477 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.148457    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c05179-576f48d86c-8kzg5"]
Sep 16 05:09:18.154079 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.154062    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:18.161963 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.161943    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"]
Sep 16 05:09:18.164003 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.163986    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c05179-5cbf97d4d8-r859h"]
Sep 16 05:09:18.257040 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.257016    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:18.786981 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.786949    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f5eaacb-14c9-42e2-beea-69e3afb989ee" path="/var/lib/kubelet/pods/3f5eaacb-14c9-42e2-beea-69e3afb989ee/volumes"
Sep 16 05:09:18.787312 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:18.787298    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c5e637-bb87-4301-9b70-65b8c3953dd0" path="/var/lib/kubelet/pods/b7c5e637-bb87-4301-9b70-65b8c3953dd0/volumes"
Sep 16 05:09:19.146680 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:19.146655    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:19.927609 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:19.927570    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:20.259121 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:20.259037    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:20.771432 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:20.771403    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:21.550694 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:21.550655    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:22.301852 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:22.301818    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:24.146898 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:24.146864    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:24.333690 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:24.333660    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:24.422112 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:24.422035    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:24.516965 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:24.516939    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:24.797044 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:24.796966    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:25.662739 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:25.662709    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:25.893387 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:25.893361    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:26.467781 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:26.467751    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:27.317563 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:27.317519    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:28.215874 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:28.215843    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:28.331720 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:28.331686    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:28.496924 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:28.496853    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:29.603959 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:29.603934    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:30.542168 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:30.542135    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:31.234953 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.234918    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"]
Sep 16 05:09:31.235472 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.235453    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c5e637-bb87-4301-9b70-65b8c3953dd0" containerName="payload-processing"
Sep 16 05:09:31.235537 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.235476    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c5e637-bb87-4301-9b70-65b8c3953dd0" containerName="payload-processing"
Sep 16 05:09:31.235611 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.235556    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5eaacb-14c9-42e2-beea-69e3afb989ee" containerName="maas-api"
Sep 16 05:09:31.235611 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.235566    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5eaacb-14c9-42e2-beea-69e3afb989ee" containerName="maas-api"
Sep 16 05:09:31.235724 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.235686    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f5eaacb-14c9-42e2-beea-69e3afb989ee" containerName="maas-api"
Sep 16 05:09:31.235724 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.235707    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c5e637-bb87-4301-9b70-65b8c3953dd0" containerName="payload-processing"
Sep 16 05:09:31.238867 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.238848    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.241013 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.240991    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-766a69c0-serving-cert\""
Sep 16 05:09:31.250010 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.249992    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"]
Sep 16 05:09:31.275064 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.275041    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:31.377949 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.377923    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:31.389760 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.389735    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-maas-api-tls\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.389858 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.389767    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20485994-cd05-490e-a1f7-790ecc75109d-openshift-service-ca\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.389858 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.389788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-metrics-certs\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.389858 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.389835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68pk\" (UniqueName: \"kubernetes.io/projected/20485994-cd05-490e-a1f7-790ecc75109d-kube-api-access-z68pk\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.468369 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.468348    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:31.490765 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.490714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-maas-api-tls\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.490765 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.490745    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20485994-cd05-490e-a1f7-790ecc75109d-openshift-service-ca\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.490917 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.490777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-metrics-certs\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.490917 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.490884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68pk\" (UniqueName: \"kubernetes.io/projected/20485994-cd05-490e-a1f7-790ecc75109d-kube-api-access-z68pk\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.491430 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.491408    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20485994-cd05-490e-a1f7-790ecc75109d-openshift-service-ca\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.493291 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.493265    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-metrics-certs\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.493371 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.493309    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-maas-api-tls\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.500113 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.500085    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68pk\" (UniqueName: \"kubernetes.io/projected/20485994-cd05-490e-a1f7-790ecc75109d-kube-api-access-z68pk\") pod \"maas-api-e2e-mt-766a69c0-76f869779-ggght\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.548895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.548873    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:31.802324 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.802198    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:31.832987 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.832925    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"]
Sep 16 05:09:31.835650 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:09:31.835590    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20485994_cd05_490e_a1f7_790ecc75109d.slice/crio-d6ffa05c177e70045aade48b1f135c35ceef4fed8a71c8bf94de68558c1b134c WatchSource:0}: Error finding container d6ffa05c177e70045aade48b1f135c35ceef4fed8a71c8bf94de68558c1b134c: Status 404 returned error can't find the container with id d6ffa05c177e70045aade48b1f135c35ceef4fed8a71c8bf94de68558c1b134c
Sep 16 05:09:31.902759 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.902731    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:31.911252 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.911224    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"]
Sep 16 05:09:31.915774 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.915757    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:09:31.917804 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.917785    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-766a69c0\""
Sep 16 05:09:31.928196 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.928174    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"]
Sep 16 05:09:31.996274 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.996252    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-plugins-config-volume\") pod \"payload-processing-e2e-mt-766a69c0-54c5798569-pm56d\" (UID: \"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d\") " pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:09:31.996414 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:31.996392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knv9\" (UniqueName: \"kubernetes.io/projected/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-kube-api-access-6knv9\") pod \"payload-processing-e2e-mt-766a69c0-54c5798569-pm56d\" (UID: \"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d\") " pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:09:32.097363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.097289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knv9\" (UniqueName: \"kubernetes.io/projected/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-kube-api-access-6knv9\") pod \"payload-processing-e2e-mt-766a69c0-54c5798569-pm56d\" (UID: \"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d\") " pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:09:32.097363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.097346    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-plugins-config-volume\") pod \"payload-processing-e2e-mt-766a69c0-54c5798569-pm56d\" (UID: \"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d\") " pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:09:32.098033 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.098014    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-plugins-config-volume\") pod \"payload-processing-e2e-mt-766a69c0-54c5798569-pm56d\" (UID: \"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d\") " pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:09:32.106508 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.106483    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knv9\" (UniqueName: \"kubernetes.io/projected/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-kube-api-access-6knv9\") pod \"payload-processing-e2e-mt-766a69c0-54c5798569-pm56d\" (UID: \"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d\") " pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:09:32.198222 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.198187    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght" event={"ID":"20485994-cd05-490e-a1f7-790ecc75109d","Type":"ContainerStarted","Data":"d6ffa05c177e70045aade48b1f135c35ceef4fed8a71c8bf94de68558c1b134c"}
Sep 16 05:09:32.226085 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.226059    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:09:32.292138 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.292105    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:32.501316 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.501271    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"]
Sep 16 05:09:32.503733 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:09:32.503693    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4303fb7e_4185_4a1d_bfbf_916e41aa8b1d.slice/crio-a167b2c6649d2b5148452cc0092397d177aad5513dcdb658eb2ffac152b1781d WatchSource:0}: Error finding container a167b2c6649d2b5148452cc0092397d177aad5513dcdb658eb2ffac152b1781d: Status 404 returned error can't find the container with id a167b2c6649d2b5148452cc0092397d177aad5513dcdb658eb2ffac152b1781d
Sep 16 05:09:32.559328 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.559296    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc" podUID="cfa51a65-31c9-4e04-b2c2-27d3053240a2" containerName="main" containerID="cri-o://4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225" gracePeriod=45
Sep 16 05:09:32.810809 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.810787    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:09:32.904857 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.904833    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tls-certs\") pod \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") "
Sep 16 05:09:32.905007 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.904892    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-dshm\") pod \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") "
Sep 16 05:09:32.905007 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.904950    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-home\") pod \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") "
Sep 16 05:09:32.905007 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.904977    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29ld9\" (UniqueName: \"kubernetes.io/projected/cfa51a65-31c9-4e04-b2c2-27d3053240a2-kube-api-access-29ld9\") pod \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") "
Sep 16 05:09:32.905171 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.905013    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tmp-dir\") pod \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") "
Sep 16 05:09:32.905171 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.905040    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-model-cache\") pod \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\" (UID: \"cfa51a65-31c9-4e04-b2c2-27d3053240a2\") "
Sep 16 05:09:32.905272 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.905230    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-home" (OuterVolumeSpecName: "home") pod "cfa51a65-31c9-4e04-b2c2-27d3053240a2" (UID: "cfa51a65-31c9-4e04-b2c2-27d3053240a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:09:32.905321 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.905286    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfa51a65-31c9-4e04-b2c2-27d3053240a2" (UID: "cfa51a65-31c9-4e04-b2c2-27d3053240a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:09:32.905377 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.905327    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfa51a65-31c9-4e04-b2c2-27d3053240a2" (UID: "cfa51a65-31c9-4e04-b2c2-27d3053240a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:09:32.905577 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.905558    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:32.905702 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.905582    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:32.905702 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.905620    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:32.907116 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.907097    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfa51a65-31c9-4e04-b2c2-27d3053240a2" (UID: "cfa51a65-31c9-4e04-b2c2-27d3053240a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:09:32.907352 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.907333    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-dshm" (OuterVolumeSpecName: "dshm") pod "cfa51a65-31c9-4e04-b2c2-27d3053240a2" (UID: "cfa51a65-31c9-4e04-b2c2-27d3053240a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:09:32.907413 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:32.907333    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa51a65-31c9-4e04-b2c2-27d3053240a2-kube-api-access-29ld9" (OuterVolumeSpecName: "kube-api-access-29ld9") pod "cfa51a65-31c9-4e04-b2c2-27d3053240a2" (UID: "cfa51a65-31c9-4e04-b2c2-27d3053240a2"). InnerVolumeSpecName "kube-api-access-29ld9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:09:33.006300 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.006249    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa51a65-31c9-4e04-b2c2-27d3053240a2-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:33.006300 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.006269    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa51a65-31c9-4e04-b2c2-27d3053240a2-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:33.006300 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.006277    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ld9\" (UniqueName: \"kubernetes.io/projected/cfa51a65-31c9-4e04-b2c2-27d3053240a2-kube-api-access-29ld9\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:33.190090 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.190067    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:33.207397 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.207362    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d" event={"ID":"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d","Type":"ContainerStarted","Data":"d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d"}
Sep 16 05:09:33.207533 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.207430    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d" event={"ID":"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d","Type":"ContainerStarted","Data":"a167b2c6649d2b5148452cc0092397d177aad5513dcdb658eb2ffac152b1781d"}
Sep 16 05:09:33.207533 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.207459    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:09:33.208878 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.208854    2573 generic.go:356] "Generic (PLEG): container finished" podID="cfa51a65-31c9-4e04-b2c2-27d3053240a2" containerID="4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225" exitCode=0
Sep 16 05:09:33.208975 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.208884    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"
Sep 16 05:09:33.208975 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.208932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc" event={"ID":"cfa51a65-31c9-4e04-b2c2-27d3053240a2","Type":"ContainerDied","Data":"4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225"}
Sep 16 05:09:33.208975 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.208965    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc" event={"ID":"cfa51a65-31c9-4e04-b2c2-27d3053240a2","Type":"ContainerDied","Data":"f494aabaa086c31156fb24e680affea919b00af6dece4bcd365d16b1d57772b4"}
Sep 16 05:09:33.209200 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.208989    2573 scope.go:117] "RemoveContainer" containerID="4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225"
Sep 16 05:09:33.211049 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.210987    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght" event={"ID":"20485994-cd05-490e-a1f7-790ecc75109d","Type":"ContainerStarted","Data":"96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9"}
Sep 16 05:09:33.211253 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.211188    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:33.218403 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.218389    2573 scope.go:117] "RemoveContainer" containerID="4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225"
Sep 16 05:09:33.218684 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:09:33.218665    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225\": container with ID starting with 4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225 not found: ID does not exist" containerID="4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225"
Sep 16 05:09:33.218747 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.218693    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225"} err="failed to get container status \"4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225\": rpc error: code = NotFound desc = could not find container \"4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225\": container with ID starting with 4053e9c25813642842bde744874243a3eb5f3ef06f89e143c726bb900675b225 not found: ID does not exist"
Sep 16 05:09:33.226251 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.226213    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d" podStartSLOduration=2.226199899 podStartE2EDuration="2.226199899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:33.22449148 +0000 UTC m=+1236.890421093" watchObservedRunningTime="2026-09-16 05:09:33.226199899 +0000 UTC m=+1236.892129494"
Sep 16 05:09:33.242720 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.242682    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght" podStartSLOduration=1.751539301 podStartE2EDuration="2.242668358s" totalImagesPullingTime="491.129057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:33.242347546 +0000 UTC m=+1236.908277138" watchObservedRunningTime="2026-09-16 05:09:33.242668358 +0000 UTC m=+1236.908597951"
Sep 16 05:09:33.258468 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.258422    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"]
Sep 16 05:09:33.261780 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.261759    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d1f291/test-model-d1f291-kserve-68f68f9cc9-xkjdc"]
Sep 16 05:09:33.845621 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:33.845568    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:34.282645 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:34.282614    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:09:34.502626 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:34.502584    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:34.787634 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:34.787583    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa51a65-31c9-4e04-b2c2-27d3053240a2" path="/var/lib/kubelet/pods/cfa51a65-31c9-4e04-b2c2-27d3053240a2/volumes"
Sep 16 05:09:35.058809 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.058728    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"]
Sep 16 05:09:35.059239 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.058915    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4" podUID="146993cb-e432-4446-94a4-75e95c8c3ad3" containerName="maas-api" containerID="cri-o://7e0e6e0867aefaf463d7eae0c9dad84d3960c6aec09e0a77a0d531a8c8090d09" gracePeriod=30
Sep 16 05:09:35.087414 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.087393    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:35.109885 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.109845    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"]
Sep 16 05:09:35.110280 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.110251    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll" podUID="fdeb454c-1e72-4c98-add6-f2dc16469300" containerName="payload-pre-processing" containerID="cri-o://1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80" gracePeriod=30
Sep 16 05:09:35.225286 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.225077    2573 generic.go:356] "Generic (PLEG): container finished" podID="146993cb-e432-4446-94a4-75e95c8c3ad3" containerID="7e0e6e0867aefaf463d7eae0c9dad84d3960c6aec09e0a77a0d531a8c8090d09" exitCode=0
Sep 16 05:09:35.225286 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.225165    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4" event={"ID":"146993cb-e432-4446-94a4-75e95c8c3ad3","Type":"ContainerDied","Data":"7e0e6e0867aefaf463d7eae0c9dad84d3960c6aec09e0a77a0d531a8c8090d09"}
Sep 16 05:09:35.356515 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.356490    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:09:35.387036 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.387013    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:09:35.529723 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.529695    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9crn7\" (UniqueName: \"kubernetes.io/projected/146993cb-e432-4446-94a4-75e95c8c3ad3-kube-api-access-9crn7\") pod \"146993cb-e432-4446-94a4-75e95c8c3ad3\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") "
Sep 16 05:09:35.529903 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.529727    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdeb454c-1e72-4c98-add6-f2dc16469300-plugins-config-volume\") pod \"fdeb454c-1e72-4c98-add6-f2dc16469300\" (UID: \"fdeb454c-1e72-4c98-add6-f2dc16469300\") "
Sep 16 05:09:35.529903 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.529763    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146993cb-e432-4446-94a4-75e95c8c3ad3-openshift-service-ca\") pod \"146993cb-e432-4446-94a4-75e95c8c3ad3\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") "
Sep 16 05:09:35.529903 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.529885    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-metrics-certs\") pod \"146993cb-e432-4446-94a4-75e95c8c3ad3\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") "
Sep 16 05:09:35.530079 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.529920    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-maas-api-tls\") pod \"146993cb-e432-4446-94a4-75e95c8c3ad3\" (UID: \"146993cb-e432-4446-94a4-75e95c8c3ad3\") "
Sep 16 05:09:35.530079 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.529977    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgvr7\" (UniqueName: \"kubernetes.io/projected/fdeb454c-1e72-4c98-add6-f2dc16469300-kube-api-access-bgvr7\") pod \"fdeb454c-1e72-4c98-add6-f2dc16469300\" (UID: \"fdeb454c-1e72-4c98-add6-f2dc16469300\") "
Sep 16 05:09:35.530184 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.530121    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146993cb-e432-4446-94a4-75e95c8c3ad3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "146993cb-e432-4446-94a4-75e95c8c3ad3" (UID: "146993cb-e432-4446-94a4-75e95c8c3ad3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:09:35.530184 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.530133    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdeb454c-1e72-4c98-add6-f2dc16469300-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdeb454c-1e72-4c98-add6-f2dc16469300" (UID: "fdeb454c-1e72-4c98-add6-f2dc16469300"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:09:35.530354 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.530330    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdeb454c-1e72-4c98-add6-f2dc16469300-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:35.530440 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.530358    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146993cb-e432-4446-94a4-75e95c8c3ad3-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:35.532063 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.532029    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "146993cb-e432-4446-94a4-75e95c8c3ad3" (UID: "146993cb-e432-4446-94a4-75e95c8c3ad3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:09:35.532165 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.532117    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146993cb-e432-4446-94a4-75e95c8c3ad3-kube-api-access-9crn7" (OuterVolumeSpecName: "kube-api-access-9crn7") pod "146993cb-e432-4446-94a4-75e95c8c3ad3" (UID: "146993cb-e432-4446-94a4-75e95c8c3ad3"). InnerVolumeSpecName "kube-api-access-9crn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:09:35.532258 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.532240    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdeb454c-1e72-4c98-add6-f2dc16469300-kube-api-access-bgvr7" (OuterVolumeSpecName: "kube-api-access-bgvr7") pod "fdeb454c-1e72-4c98-add6-f2dc16469300" (UID: "fdeb454c-1e72-4c98-add6-f2dc16469300"). InnerVolumeSpecName "kube-api-access-bgvr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:09:35.532360 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.532343    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "146993cb-e432-4446-94a4-75e95c8c3ad3" (UID: "146993cb-e432-4446-94a4-75e95c8c3ad3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:09:35.631123 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.631101    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9crn7\" (UniqueName: \"kubernetes.io/projected/146993cb-e432-4446-94a4-75e95c8c3ad3-kube-api-access-9crn7\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:35.631123 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.631121    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:35.631262 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.631130    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146993cb-e432-4446-94a4-75e95c8c3ad3-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:35.631262 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:35.631139    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgvr7\" (UniqueName: \"kubernetes.io/projected/fdeb454c-1e72-4c98-add6-f2dc16469300-kube-api-access-bgvr7\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:09:36.231876 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.231844    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"
Sep 16 05:09:36.232277 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.231874    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4" event={"ID":"146993cb-e432-4446-94a4-75e95c8c3ad3","Type":"ContainerDied","Data":"2d012173df1ac2f749def4a6778beb2b46a47c1ae4daa6d7161238ef30fee0b2"}
Sep 16 05:09:36.232277 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.231940    2573 scope.go:117] "RemoveContainer" containerID="7e0e6e0867aefaf463d7eae0c9dad84d3960c6aec09e0a77a0d531a8c8090d09"
Sep 16 05:09:36.233349 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.233328    2573 generic.go:356] "Generic (PLEG): container finished" podID="fdeb454c-1e72-4c98-add6-f2dc16469300" containerID="1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80" exitCode=0
Sep 16 05:09:36.233423 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.233383    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"
Sep 16 05:09:36.233423 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.233402    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll" event={"ID":"fdeb454c-1e72-4c98-add6-f2dc16469300","Type":"ContainerDied","Data":"1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80"}
Sep 16 05:09:36.233535 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.233454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll" event={"ID":"fdeb454c-1e72-4c98-add6-f2dc16469300","Type":"ContainerDied","Data":"f245d6419f9dd41932cbe1660ef744ee94b7f30466b9a27fb0ae64cb2a99852e"}
Sep 16 05:09:36.242268 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.242249    2573 scope.go:117] "RemoveContainer" containerID="1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80"
Sep 16 05:09:36.256108 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.256089    2573 scope.go:117] "RemoveContainer" containerID="1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80"
Sep 16 05:09:36.256408 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:09:36.256388    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80\": container with ID starting with 1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80 not found: ID does not exist" containerID="1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80"
Sep 16 05:09:36.256476 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.256418    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80"} err="failed to get container status \"1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80\": rpc error: code = NotFound desc = could not find container \"1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80\": container with ID starting with 1ba63282ae256c862efced541525fcf07a237f287722ceaa18ca29108bc4be80 not found: ID does not exist"
Sep 16 05:09:36.257523 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.257502    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"]
Sep 16 05:09:36.259794 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.259773    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d1f291-685b58794f-99js4"]
Sep 16 05:09:36.268937 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.268917    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"]
Sep 16 05:09:36.273385 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.273359    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d1f291-7c77b5c9db-2dvll"]
Sep 16 05:09:36.722266 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.722215    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:36.786943 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.786917    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="146993cb-e432-4446-94a4-75e95c8c3ad3" path="/var/lib/kubelet/pods/146993cb-e432-4446-94a4-75e95c8c3ad3/volumes"
Sep 16 05:09:36.787278 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:36.787265    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdeb454c-1e72-4c98-add6-f2dc16469300" path="/var/lib/kubelet/pods/fdeb454c-1e72-4c98-add6-f2dc16469300/volumes"
Sep 16 05:09:37.337206 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:37.337178    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:38.027390 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.027345    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:38.742313 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.742283    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"]
Sep 16 05:09:38.742682 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.742649    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146993cb-e432-4446-94a4-75e95c8c3ad3" containerName="maas-api"
Sep 16 05:09:38.742682 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.742661    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="146993cb-e432-4446-94a4-75e95c8c3ad3" containerName="maas-api"
Sep 16 05:09:38.742682 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.742681    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdeb454c-1e72-4c98-add6-f2dc16469300" containerName="payload-pre-processing"
Sep 16 05:09:38.742796 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.742686    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdeb454c-1e72-4c98-add6-f2dc16469300" containerName="payload-pre-processing"
Sep 16 05:09:38.742796 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.742701    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa51a65-31c9-4e04-b2c2-27d3053240a2" containerName="main"
Sep 16 05:09:38.742796 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.742707    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa51a65-31c9-4e04-b2c2-27d3053240a2" containerName="main"
Sep 16 05:09:38.742796 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.742760    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa51a65-31c9-4e04-b2c2-27d3053240a2" containerName="main"
Sep 16 05:09:38.742796 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.742769    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdeb454c-1e72-4c98-add6-f2dc16469300" containerName="payload-pre-processing"
Sep 16 05:09:38.742796 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.742777    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="146993cb-e432-4446-94a4-75e95c8c3ad3" containerName="maas-api"
Sep 16 05:09:38.747192 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.747174    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.750515 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.750495    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"default-dockercfg-j9d82\""
Sep 16 05:09:38.751192 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.751164    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"kube-root-ca.crt\""
Sep 16 05:09:38.751192 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.751184    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"openshift-service-ca.crt\""
Sep 16 05:09:38.751362 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.751205    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"e2e-conflict-b-2a88c17b-kserve-self-signed-certs\""
Sep 16 05:09:38.771525 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.771500    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"]
Sep 16 05:09:38.847094 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.847063    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:38.854223 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.854202    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-home\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.854294 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.854237    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzg82\" (UniqueName: \"kubernetes.io/projected/bab541c9-7ff5-4db4-8238-820a4a518902-kube-api-access-hzg82\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.854294 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.854269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-tmp-dir\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.854373 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.854350    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab541c9-7ff5-4db4-8238-820a4a518902-tls-certs\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.854410 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.854392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-dshm\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.854479 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.854431    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-model-cache\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.950233 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.950208    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:38.955582 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.955562    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-model-cache\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.955664 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.955607    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-home\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.955664 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.955629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzg82\" (UniqueName: \"kubernetes.io/projected/bab541c9-7ff5-4db4-8238-820a4a518902-kube-api-access-hzg82\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.955664 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.955647    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-tmp-dir\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.955810 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.955689    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab541c9-7ff5-4db4-8238-820a4a518902-tls-certs\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.955810 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.955733    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-dshm\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.955966 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.955945    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-model-cache\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.956037 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.956011    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-home\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.956086 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.956041    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-tmp-dir\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.957928 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.957911    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-dshm\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.958238 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.958218    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab541c9-7ff5-4db4-8238-820a4a518902-tls-certs\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:38.964770 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:38.964748    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzg82\" (UniqueName: \"kubernetes.io/projected/bab541c9-7ff5-4db4-8238-820a4a518902-kube-api-access-hzg82\") pod \"e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:39.055926 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:39.055870    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:39.223282 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:39.223256    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:09:39.329154 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:39.329084    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"]
Sep 16 05:09:39.332269 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:09:39.332240    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbab541c9_7ff5_4db4_8238_820a4a518902.slice/crio-75262c45cf8a232b986228bb3e198aa66fed6a7f4b7fe3841ca3a055557830a3 WatchSource:0}: Error finding container 75262c45cf8a232b986228bb3e198aa66fed6a7f4b7fe3841ca3a055557830a3: Status 404 returned error can't find the container with id 75262c45cf8a232b986228bb3e198aa66fed6a7f4b7fe3841ca3a055557830a3
Sep 16 05:09:39.571371 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:39.571340    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:40.194027 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:40.193997    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:40.265402 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:40.265363    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr" event={"ID":"bab541c9-7ff5-4db4-8238-820a4a518902","Type":"ContainerStarted","Data":"3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99"}
Sep 16 05:09:40.265560 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:40.265413    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr" event={"ID":"bab541c9-7ff5-4db4-8238-820a4a518902","Type":"ContainerStarted","Data":"75262c45cf8a232b986228bb3e198aa66fed6a7f4b7fe3841ca3a055557830a3"}
Sep 16 05:09:40.298446 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:40.298397    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr" podStartSLOduration=2.298382219 podStartE2EDuration="2.298382219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:40.29548892 +0000 UTC m=+1243.961418538" watchObservedRunningTime="2026-09-16 05:09:40.298382219 +0000 UTC m=+1243.964311839"
Sep 16 05:09:40.454137 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:40.454064    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:40.580912 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:40.580885    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:42.172442 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:42.172410    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:42.276644 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:42.276616    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:42.456764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:42.456693    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:43.032844 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:43.032814    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:43.393146 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:43.393115    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:43.808510 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:43.808422    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:43.943305 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:43.943273    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:44.219529 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:44.219494    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:09:45.634718 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:45.634665    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:45.843107 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:45.843077    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:46.577284 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:46.577243    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:46.716145 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:46.716106    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:46.919962 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:46.919931    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:48.197527 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:48.197488    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:49.057073 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:49.057031    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:49.057372 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:49.057344    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:49.070114 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:49.070084    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:49.254126 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:49.254093    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:49.321928 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:49.321833    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:09:50.054309 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:50.054275    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:50.866150 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:50.866114    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:51.954985 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:51.954947    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"]
Sep 16 05:09:51.958664 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:51.958644    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:51.961401 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:51.961377    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-3784e2-serving-cert\""
Sep 16 05:09:51.973090 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:51.973033    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:51.973259 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:51.973191    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:51.973334 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:51.973267    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:51.973334 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:51.973314    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcdbq\" (UniqueName: \"kubernetes.io/projected/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-kube-api-access-bcdbq\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:51.976201 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:51.976176    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"]
Sep 16 05:09:52.032529 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.032496    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:52.074411 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.074382    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:52.074634 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.074424    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcdbq\" (UniqueName: \"kubernetes.io/projected/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-kube-api-access-bcdbq\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:52.074634 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.074466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:52.074634 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.074500    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:52.074818 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:09:52.074647    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-serving-cert: secret "maas-api-e2e-shared-a-w3-3784e2-serving-cert" not found
Sep 16 05:09:52.074818 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:09:52.074738    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-maas-api-tls podName:9ef0e388-5c57-4d87-a2f0-d8d0d70f2137 nodeName:}" failed. No retries permitted until 2026-09-16 05:09:52.574705686 +0000 UTC m=+1256.240635275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-maas-api-tls") pod "maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47" (UID: "9ef0e388-5c57-4d87-a2f0-d8d0d70f2137") : secret "maas-api-e2e-shared-a-w3-3784e2-serving-cert" not found
Sep 16 05:09:52.075267 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.075240    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:52.077099 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.077076    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:52.089092 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.089066    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcdbq\" (UniqueName: \"kubernetes.io/projected/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-kube-api-access-bcdbq\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:52.138588 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.138554    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:52.236350 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.236253    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:52.579785 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.579684    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:52.582260 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.582235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:52.864024 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.863936    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:52.869022 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:52.868992    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:53.001609 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:53.001560    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:53.150103 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:53.150049    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"]
Sep 16 05:09:53.152959 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:09:53.152930    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef0e388_5c57_4d87_a2f0_d8d0d70f2137.slice/crio-92495cc718b760f9817d6b91f8536843ff8dfdb2d30838ce7e85a29a15e70a91 WatchSource:0}: Error finding container 92495cc718b760f9817d6b91f8536843ff8dfdb2d30838ce7e85a29a15e70a91: Status 404 returned error can't find the container with id 92495cc718b760f9817d6b91f8536843ff8dfdb2d30838ce7e85a29a15e70a91
Sep 16 05:09:53.218654 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:53.218621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:53.338328 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:53.338296    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47" event={"ID":"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137","Type":"ContainerStarted","Data":"92495cc718b760f9817d6b91f8536843ff8dfdb2d30838ce7e85a29a15e70a91"}
Sep 16 05:09:53.394062 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:53.394022    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:54.344436 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:54.344404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47" event={"ID":"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137","Type":"ContainerStarted","Data":"60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509"}
Sep 16 05:09:54.344894 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:54.344479    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:09:54.371541 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:54.371490    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47" podStartSLOduration=3.053182624 podStartE2EDuration="3.371471563s" totalImagesPullingTime="318.288939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:54.369882389 +0000 UTC m=+1258.035811981" watchObservedRunningTime="2026-09-16 05:09:54.371471563 +0000 UTC m=+1258.037401156"
Sep 16 05:09:54.488804 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:54.488771    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:55.168115 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:55.168084    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:09:55.924689 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:09:55.924658    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:00.356382 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:00.356349    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:10:07.789073 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:07.789036    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:10:11.737408 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:11.737374    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:14.886411 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:14.886371    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:15.640504 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:15.640470    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:16.488524 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:16.488494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:16.588923 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:16.588883    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:16.826686 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:16.826576    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:17.312559 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:17.312526    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:17.397417 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:17.397387    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:18.169853 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:18.169825    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:18.967834 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:18.967796    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:20.074796 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:20.074753    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:20.575263 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:20.575235    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:20.669501 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:20.669473    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:20.953026 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:20.952993    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:21.771237 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:21.771206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:22.485450 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:22.485419    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:23.331188 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:23.331155    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:23.750246 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:23.750211    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:24.227474 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:24.227447    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:24.932721 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:24.932682    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:25.647816 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:25.647784    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:26.458111 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:26.458083    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:26.561668 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:26.561636    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:26.934531 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:26.934495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:27.028218 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.028186    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:27.032887 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.032858    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"]
Sep 16 05:10:27.036304 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.036269    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:10:27.039101 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.039073    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-10ee79-dockercfg-nxrcp\""
Sep 16 05:10:27.039512 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.039448    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-10ee79\""
Sep 16 05:10:27.049147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.049121    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"]
Sep 16 05:10:27.053808 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.053783    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"]
Sep 16 05:10:27.057009 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.056987    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:10:27.073817 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.073788    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"]
Sep 16 05:10:27.098298 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.098262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62\" (UID: \"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:10:27.098475 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.098303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z67nw\" (UniqueName: \"kubernetes.io/projected/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-kube-api-access-z67nw\") pod \"payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62\" (UID: \"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:10:27.098475 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.098403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758db869-4c2f-4443-a8ff-89a913aa741d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7\" (UID: \"758db869-4c2f-4443-a8ff-89a913aa741d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:10:27.098475 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.098463    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wdgf\" (UniqueName: \"kubernetes.io/projected/758db869-4c2f-4443-a8ff-89a913aa741d-kube-api-access-4wdgf\") pod \"payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7\" (UID: \"758db869-4c2f-4443-a8ff-89a913aa741d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:10:27.130623 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.130560    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:27.199203 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.199106    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62\" (UID: \"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:10:27.199203 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.199146    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z67nw\" (UniqueName: \"kubernetes.io/projected/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-kube-api-access-z67nw\") pod \"payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62\" (UID: \"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:10:27.199203 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.199184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758db869-4c2f-4443-a8ff-89a913aa741d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7\" (UID: \"758db869-4c2f-4443-a8ff-89a913aa741d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:10:27.199492 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.199228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wdgf\" (UniqueName: \"kubernetes.io/projected/758db869-4c2f-4443-a8ff-89a913aa741d-kube-api-access-4wdgf\") pod \"payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7\" (UID: \"758db869-4c2f-4443-a8ff-89a913aa741d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:10:27.199886 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.199865    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62\" (UID: \"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:10:27.199960 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.199907    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758db869-4c2f-4443-a8ff-89a913aa741d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7\" (UID: \"758db869-4c2f-4443-a8ff-89a913aa741d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:10:27.207197 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.207175    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z67nw\" (UniqueName: \"kubernetes.io/projected/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-kube-api-access-z67nw\") pod \"payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62\" (UID: \"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:10:27.207383 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.207364    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wdgf\" (UniqueName: \"kubernetes.io/projected/758db869-4c2f-4443-a8ff-89a913aa741d-kube-api-access-4wdgf\") pod \"payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7\" (UID: \"758db869-4c2f-4443-a8ff-89a913aa741d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:10:27.341306 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.341271    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:27.348216 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.348196    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:10:27.367315 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.367285    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:10:27.628340 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.628313    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"]
Sep 16 05:10:27.630528 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:10:27.630500    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758db869_4c2f_4443_a8ff_89a913aa741d.slice/crio-d1a832572f4b76bd4c29a86995eade083df691108cf2e46c2d826b82d56f422d WatchSource:0}: Error finding container d1a832572f4b76bd4c29a86995eade083df691108cf2e46c2d826b82d56f422d: Status 404 returned error can't find the container with id d1a832572f4b76bd4c29a86995eade083df691108cf2e46c2d826b82d56f422d
Sep 16 05:10:27.661924 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:27.661898    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"]
Sep 16 05:10:27.665770 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:10:27.665739    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f64c4f_39a1_4090_a2a3_c7bbb896b2e0.slice/crio-6fc9931197373a9ce6c27981b8ee0b6c549c6f5eac79c3709c9294fc8578a81a WatchSource:0}: Error finding container 6fc9931197373a9ce6c27981b8ee0b6c549c6f5eac79c3709c9294fc8578a81a: Status 404 returned error can't find the container with id 6fc9931197373a9ce6c27981b8ee0b6c549c6f5eac79c3709c9294fc8578a81a
Sep 16 05:10:28.153759 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:28.153725    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:28.511527 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:28.511432    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62" event={"ID":"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0","Type":"ContainerStarted","Data":"89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3"}
Sep 16 05:10:28.511527 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:28.511496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62" event={"ID":"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0","Type":"ContainerStarted","Data":"6fc9931197373a9ce6c27981b8ee0b6c549c6f5eac79c3709c9294fc8578a81a"}
Sep 16 05:10:28.511796 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:28.511531    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:10:28.513048 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:28.513024    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7" event={"ID":"758db869-4c2f-4443-a8ff-89a913aa741d","Type":"ContainerStarted","Data":"f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c"}
Sep 16 05:10:28.513150 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:28.513066    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7" event={"ID":"758db869-4c2f-4443-a8ff-89a913aa741d","Type":"ContainerStarted","Data":"d1a832572f4b76bd4c29a86995eade083df691108cf2e46c2d826b82d56f422d"}
Sep 16 05:10:28.513150 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:28.513089    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:10:28.531733 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:28.531679    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62" podStartSLOduration=1.5316602430000001 podStartE2EDuration="1.531660243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:10:28.527007701 +0000 UTC m=+1292.192937294" watchObservedRunningTime="2026-09-16 05:10:28.531660243 +0000 UTC m=+1292.197589837"
Sep 16 05:10:28.548386 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:28.548335    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7" podStartSLOduration=1.548320938 podStartE2EDuration="1.548320938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:10:28.546735668 +0000 UTC m=+1292.212665260" watchObservedRunningTime="2026-09-16 05:10:28.548320938 +0000 UTC m=+1292.214250531"
Sep 16 05:10:31.989073 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:31.989039    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:10:39.519691 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:39.519658    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:10:39.520159 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:39.519732    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:10:48.454144 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:48.454110    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:48.559252 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:48.559226    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:49.145101 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:49.145072    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:49.874173 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:49.874104    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:50.630398 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:50.630365    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:50.730373 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:50.730313    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:51.543552 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:51.543524    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:52.263968 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:52.263937    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:53.161756 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:53.161723    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:10:53.858397 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:10:53.858370    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:01.279459 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:01.279426    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:11:09.859135 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:09.859101    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"]
Sep 16 05:11:09.887374 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:09.887343    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:24.878531 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:24.878433    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr" podUID="bab541c9-7ff5-4db4-8238-820a4a518902" containerName="main" containerID="cri-o://3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99" gracePeriod=45
Sep 16 05:11:25.200110 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.200075    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:11:25.290439 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290409    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-dshm\") pod \"bab541c9-7ff5-4db4-8238-820a4a518902\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") "
Sep 16 05:11:25.290662 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290454    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzg82\" (UniqueName: \"kubernetes.io/projected/bab541c9-7ff5-4db4-8238-820a4a518902-kube-api-access-hzg82\") pod \"bab541c9-7ff5-4db4-8238-820a4a518902\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") "
Sep 16 05:11:25.290662 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290479    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-model-cache\") pod \"bab541c9-7ff5-4db4-8238-820a4a518902\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") "
Sep 16 05:11:25.290662 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290525    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-tmp-dir\") pod \"bab541c9-7ff5-4db4-8238-820a4a518902\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") "
Sep 16 05:11:25.290662 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290550    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab541c9-7ff5-4db4-8238-820a4a518902-tls-certs\") pod \"bab541c9-7ff5-4db4-8238-820a4a518902\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") "
Sep 16 05:11:25.290662 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290584    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-home\") pod \"bab541c9-7ff5-4db4-8238-820a4a518902\" (UID: \"bab541c9-7ff5-4db4-8238-820a4a518902\") "
Sep 16 05:11:25.290937 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290840    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-model-cache" (OuterVolumeSpecName: "model-cache") pod "bab541c9-7ff5-4db4-8238-820a4a518902" (UID: "bab541c9-7ff5-4db4-8238-820a4a518902"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:11:25.290937 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290851    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bab541c9-7ff5-4db4-8238-820a4a518902" (UID: "bab541c9-7ff5-4db4-8238-820a4a518902"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:11:25.291015 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290951    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:11:25.291015 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290972    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:11:25.291015 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.290942    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-home" (OuterVolumeSpecName: "home") pod "bab541c9-7ff5-4db4-8238-820a4a518902" (UID: "bab541c9-7ff5-4db4-8238-820a4a518902"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:11:25.293246 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.293205    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-dshm" (OuterVolumeSpecName: "dshm") pod "bab541c9-7ff5-4db4-8238-820a4a518902" (UID: "bab541c9-7ff5-4db4-8238-820a4a518902"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:11:25.293383 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.293344    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab541c9-7ff5-4db4-8238-820a4a518902-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bab541c9-7ff5-4db4-8238-820a4a518902" (UID: "bab541c9-7ff5-4db4-8238-820a4a518902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:11:25.293545 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.293519    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab541c9-7ff5-4db4-8238-820a4a518902-kube-api-access-hzg82" (OuterVolumeSpecName: "kube-api-access-hzg82") pod "bab541c9-7ff5-4db4-8238-820a4a518902" (UID: "bab541c9-7ff5-4db4-8238-820a4a518902"). InnerVolumeSpecName "kube-api-access-hzg82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:11:25.392447 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.392402    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:11:25.392447 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.392437    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzg82\" (UniqueName: \"kubernetes.io/projected/bab541c9-7ff5-4db4-8238-820a4a518902-kube-api-access-hzg82\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:11:25.392447 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.392448    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab541c9-7ff5-4db4-8238-820a4a518902-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:11:25.392447 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.392457    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab541c9-7ff5-4db4-8238-820a4a518902-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:11:25.806488 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.806446    2573 generic.go:356] "Generic (PLEG): container finished" podID="bab541c9-7ff5-4db4-8238-820a4a518902" containerID="3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99" exitCode=0
Sep 16 05:11:25.806488 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.806488    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"
Sep 16 05:11:25.806823 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.806525    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr" event={"ID":"bab541c9-7ff5-4db4-8238-820a4a518902","Type":"ContainerDied","Data":"3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99"}
Sep 16 05:11:25.806823 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.806579    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr" event={"ID":"bab541c9-7ff5-4db4-8238-820a4a518902","Type":"ContainerDied","Data":"75262c45cf8a232b986228bb3e198aa66fed6a7f4b7fe3841ca3a055557830a3"}
Sep 16 05:11:25.806823 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.806617    2573 scope.go:117] "RemoveContainer" containerID="3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99"
Sep 16 05:11:25.817094 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.817064    2573 scope.go:117] "RemoveContainer" containerID="3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99"
Sep 16 05:11:25.817525 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:11:25.817497    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99\": container with ID starting with 3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99 not found: ID does not exist" containerID="3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99"
Sep 16 05:11:25.817644 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.817538    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99"} err="failed to get container status \"3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99\": rpc error: code = NotFound desc = could not find container \"3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99\": container with ID starting with 3ded19fdbe3dbc279bf5efa70f53b18413e8ab13e01b546bfbe73fda90730c99 not found: ID does not exist"
Sep 16 05:11:25.830116 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.830078    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"]
Sep 16 05:11:25.835886 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:25.835840    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-b-2a88c17b-kserve-54bfdfd7cf-pxddr"]
Sep 16 05:11:26.791359 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:26.791321    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bab541c9-7ff5-4db4-8238-820a4a518902" path="/var/lib/kubelet/pods/bab541c9-7ff5-4db4-8238-820a4a518902/volumes"
Sep 16 05:11:28.188059 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:28.188016    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:11:35.143678 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.143630    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"]
Sep 16 05:11:35.144270 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.144254    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab541c9-7ff5-4db4-8238-820a4a518902" containerName="main"
Sep 16 05:11:35.144332 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.144274    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab541c9-7ff5-4db4-8238-820a4a518902" containerName="main"
Sep 16 05:11:35.144429 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.144418    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bab541c9-7ff5-4db4-8238-820a4a518902" containerName="main"
Sep 16 05:11:35.149980 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.149953    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.153459 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.153420    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3784e2\"/\"kube-root-ca.crt\""
Sep 16 05:11:35.154060 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.154004    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3784e2\"/\"auth-test-model-3784e2-kserve-self-signed-certs\""
Sep 16 05:11:35.154147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.154111    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3784e2\"/\"default-dockercfg-zr4fr\""
Sep 16 05:11:35.154203 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.154188    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3784e2\"/\"openshift-service-ca.crt\""
Sep 16 05:11:35.165470 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.165439    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"]
Sep 16 05:11:35.199447 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.199383    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c97f6e-5742-4f0d-b6aa-826e35634677-tls-certs\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.199814 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.199467    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-model-cache\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.199814 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.199507    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-dshm\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.199814 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.199636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-home\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.199814 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.199710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zpkb\" (UniqueName: \"kubernetes.io/projected/12c97f6e-5742-4f0d-b6aa-826e35634677-kube-api-access-5zpkb\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.199814 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.199759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-tmp-dir\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.220840 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.220795    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:35.300690 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.300647    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zpkb\" (UniqueName: \"kubernetes.io/projected/12c97f6e-5742-4f0d-b6aa-826e35634677-kube-api-access-5zpkb\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.300909 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.300702    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-tmp-dir\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.300909 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.300777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c97f6e-5742-4f0d-b6aa-826e35634677-tls-certs\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.300909 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.300815    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-model-cache\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.300909 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.300855    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-dshm\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.300909 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.300898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-home\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.301176 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.301146    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-tmp-dir\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.301242 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.301197    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-model-cache\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.301328 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.301309    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-home\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.303446 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.303414    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-dshm\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.303715 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.303697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c97f6e-5742-4f0d-b6aa-826e35634677-tls-certs\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.315894 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.315864    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zpkb\" (UniqueName: \"kubernetes.io/projected/12c97f6e-5742-4f0d-b6aa-826e35634677-kube-api-access-5zpkb\") pod \"auth-test-model-3784e2-kserve-64d55f59b7-wq5l9\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.461067 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.460963    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:35.739719 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.739689    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"]
Sep 16 05:11:35.741694 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:11:35.741664    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12c97f6e_5742_4f0d_b6aa_826e35634677.slice/crio-64afe26c52e32d0448e397d161777ecc54ae4f6340cea6fa5eeaeb8b06941b5c WatchSource:0}: Error finding container 64afe26c52e32d0448e397d161777ecc54ae4f6340cea6fa5eeaeb8b06941b5c: Status 404 returned error can't find the container with id 64afe26c52e32d0448e397d161777ecc54ae4f6340cea6fa5eeaeb8b06941b5c
Sep 16 05:11:35.863160 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.863113    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9" event={"ID":"12c97f6e-5742-4f0d-b6aa-826e35634677","Type":"ContainerStarted","Data":"489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0"}
Sep 16 05:11:35.863338 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.863173    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9" event={"ID":"12c97f6e-5742-4f0d-b6aa-826e35634677","Type":"ContainerStarted","Data":"64afe26c52e32d0448e397d161777ecc54ae4f6340cea6fa5eeaeb8b06941b5c"}
Sep 16 05:11:35.882571 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:35.882492    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9" podStartSLOduration=0.882465851 podStartE2EDuration="882.465851ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:11:35.878775268 +0000 UTC m=+1359.544704885" watchObservedRunningTime="2026-09-16 05:11:35.882465851 +0000 UTC m=+1359.548395445"
Sep 16 05:11:38.252151 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:38.252113    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:45.461149 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:45.461093    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:45.461560 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:45.461171    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:45.474043 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:45.474011    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:45.926747 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:45.926714    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:11:55.112204 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:55.112164    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:55.836656 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:55.836626    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:56.491785 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:56.491754    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:57.115001 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:57.114967    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:57.846500 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:57.846467    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:58.137902 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.137868    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"]
Sep 16 05:11:58.141500 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.141479    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.143434 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.143411    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"kube-root-ca.crt\""
Sep 16 05:11:58.144012 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.143989    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"distinct-w4-12b6c6-kserve-self-signed-certs\""
Sep 16 05:11:58.144116 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.144026    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"default-dockercfg-j9d82\""
Sep 16 05:11:58.144116 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.144098    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"openshift-service-ca.crt\""
Sep 16 05:11:58.152528 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.152506    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"]
Sep 16 05:11:58.159809 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.159783    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:58.323181 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.323139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a34a03d-c457-4a5b-aba9-84fb634e469c-tls-certs\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.323181 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.323185    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-dshm\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.323464 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.323205    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svldr\" (UniqueName: \"kubernetes.io/projected/4a34a03d-c457-4a5b-aba9-84fb634e469c-kube-api-access-svldr\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.323464 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.323330    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-model-cache\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.323464 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.323411    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-tmp-dir\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.323464 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.323446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-home\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.424869 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.424779    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-model-cache\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.424869 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.424828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-tmp-dir\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.424869 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.424849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-home\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.425159 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.424903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a34a03d-c457-4a5b-aba9-84fb634e469c-tls-certs\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.425159 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.424922    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-dshm\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.425159 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.424938    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svldr\" (UniqueName: \"kubernetes.io/projected/4a34a03d-c457-4a5b-aba9-84fb634e469c-kube-api-access-svldr\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.425333 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.425238    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-model-cache\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.425333 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.425307    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-tmp-dir\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.425333 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.425322    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-home\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.427273 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.427251    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-dshm\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.427574 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.427553    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a34a03d-c457-4a5b-aba9-84fb634e469c-tls-certs\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.432457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.432431    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svldr\" (UniqueName: \"kubernetes.io/projected/4a34a03d-c457-4a5b-aba9-84fb634e469c-kube-api-access-svldr\") pod \"distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") " pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.451090 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.451056    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:11:58.495267 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.495235    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:58.729166 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.729140    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"]
Sep 16 05:11:58.731827 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:11:58.731793    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a34a03d_c457_4a5b_aba9_84fb634e469c.slice/crio-5488387355fd28dbf11521bd08bce569084add5e6deaf32036401bd8d519366d WatchSource:0}: Error finding container 5488387355fd28dbf11521bd08bce569084add5e6deaf32036401bd8d519366d: Status 404 returned error can't find the container with id 5488387355fd28dbf11521bd08bce569084add5e6deaf32036401bd8d519366d
Sep 16 05:11:58.983896 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.983804    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n" event={"ID":"4a34a03d-c457-4a5b-aba9-84fb634e469c","Type":"ContainerStarted","Data":"d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7"}
Sep 16 05:11:58.983896 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:58.983848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n" event={"ID":"4a34a03d-c457-4a5b-aba9-84fb634e469c","Type":"ContainerStarted","Data":"5488387355fd28dbf11521bd08bce569084add5e6deaf32036401bd8d519366d"}
Sep 16 05:11:59.002350 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:59.002293    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n" podStartSLOduration=1.002276873 podStartE2EDuration="1.002276873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:11:59.001469547 +0000 UTC m=+1382.667399189" watchObservedRunningTime="2026-09-16 05:11:59.002276873 +0000 UTC m=+1382.668206466"
Sep 16 05:11:59.223133 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:59.223101    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:11:59.831795 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:11:59.831748    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:00.457573 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:00.457538    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:08.397501 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:08.397462    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:12:08.451485 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:08.451441    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:12:08.451667 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:08.451502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:12:08.464169 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:08.464144    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:12:09.045824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:09.045792    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:12:10.489778 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.489735    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"]
Sep 16 05:12:10.490194 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.489948    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght" podUID="20485994-cd05-490e-a1f7-790ecc75109d" containerName="maas-api" containerID="cri-o://96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9" gracePeriod=30
Sep 16 05:12:10.506676 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.506647    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:10.569486 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.569449    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"]
Sep 16 05:12:10.569718 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.569693    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d" podUID="4303fb7e-4185-4a1d-bfbf-916e41aa8b1d" containerName="payload-processing" containerID="cri-o://d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d" gracePeriod=30
Sep 16 05:12:10.610231 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.610204    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:10.710532 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.710500    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:10.818724 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.818686    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:10.826425 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.826400    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:12:10.854883 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.854694    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:12:10.938497 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.938463    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-plugins-config-volume\") pod \"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d\" (UID: \"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d\") "
Sep 16 05:12:10.938497 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.938504    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z68pk\" (UniqueName: \"kubernetes.io/projected/20485994-cd05-490e-a1f7-790ecc75109d-kube-api-access-z68pk\") pod \"20485994-cd05-490e-a1f7-790ecc75109d\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") "
Sep 16 05:12:10.938742 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.938561    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6knv9\" (UniqueName: \"kubernetes.io/projected/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-kube-api-access-6knv9\") pod \"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d\" (UID: \"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d\") "
Sep 16 05:12:10.938742 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.938621    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-metrics-certs\") pod \"20485994-cd05-490e-a1f7-790ecc75109d\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") "
Sep 16 05:12:10.938742 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.938649    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-maas-api-tls\") pod \"20485994-cd05-490e-a1f7-790ecc75109d\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") "
Sep 16 05:12:10.938742 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.938674    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20485994-cd05-490e-a1f7-790ecc75109d-openshift-service-ca\") pod \"20485994-cd05-490e-a1f7-790ecc75109d\" (UID: \"20485994-cd05-490e-a1f7-790ecc75109d\") "
Sep 16 05:12:10.939117 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.938933    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4303fb7e-4185-4a1d-bfbf-916e41aa8b1d" (UID: "4303fb7e-4185-4a1d-bfbf-916e41aa8b1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:12:10.939117 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.939104    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20485994-cd05-490e-a1f7-790ecc75109d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20485994-cd05-490e-a1f7-790ecc75109d" (UID: "20485994-cd05-490e-a1f7-790ecc75109d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:12:10.940921 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.940897    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20485994-cd05-490e-a1f7-790ecc75109d" (UID: "20485994-cd05-490e-a1f7-790ecc75109d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:12:10.940921 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.940917    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "20485994-cd05-490e-a1f7-790ecc75109d" (UID: "20485994-cd05-490e-a1f7-790ecc75109d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:12:10.941193 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.941174    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-kube-api-access-6knv9" (OuterVolumeSpecName: "kube-api-access-6knv9") pod "4303fb7e-4185-4a1d-bfbf-916e41aa8b1d" (UID: "4303fb7e-4185-4a1d-bfbf-916e41aa8b1d"). InnerVolumeSpecName "kube-api-access-6knv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:12:10.941251 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:10.941184    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20485994-cd05-490e-a1f7-790ecc75109d-kube-api-access-z68pk" (OuterVolumeSpecName: "kube-api-access-z68pk") pod "20485994-cd05-490e-a1f7-790ecc75109d" (UID: "20485994-cd05-490e-a1f7-790ecc75109d"). InnerVolumeSpecName "kube-api-access-z68pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:12:11.039764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.039683    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68pk\" (UniqueName: \"kubernetes.io/projected/20485994-cd05-490e-a1f7-790ecc75109d-kube-api-access-z68pk\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:12:11.039764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.039711    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6knv9\" (UniqueName: \"kubernetes.io/projected/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-kube-api-access-6knv9\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:12:11.039764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.039723    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:12:11.039764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.039732    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20485994-cd05-490e-a1f7-790ecc75109d-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:12:11.039764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.039740    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20485994-cd05-490e-a1f7-790ecc75109d-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:12:11.039764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.039749    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:12:11.047254 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.047214    2573 generic.go:356] "Generic (PLEG): container finished" podID="20485994-cd05-490e-a1f7-790ecc75109d" containerID="96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9" exitCode=0
Sep 16 05:12:11.047254 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.047244    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght" event={"ID":"20485994-cd05-490e-a1f7-790ecc75109d","Type":"ContainerDied","Data":"96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9"}
Sep 16 05:12:11.047457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.047295    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght" event={"ID":"20485994-cd05-490e-a1f7-790ecc75109d","Type":"ContainerDied","Data":"d6ffa05c177e70045aade48b1f135c35ceef4fed8a71c8bf94de68558c1b134c"}
Sep 16 05:12:11.047457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.047228    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"
Sep 16 05:12:11.047457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.047321    2573 scope.go:117] "RemoveContainer" containerID="96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9"
Sep 16 05:12:11.048723 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.048692    2573 generic.go:356] "Generic (PLEG): container finished" podID="4303fb7e-4185-4a1d-bfbf-916e41aa8b1d" containerID="d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d" exitCode=0
Sep 16 05:12:11.048827 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.048736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d" event={"ID":"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d","Type":"ContainerDied","Data":"d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d"}
Sep 16 05:12:11.048827 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.048745    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"
Sep 16 05:12:11.048827 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.048767    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d" event={"ID":"4303fb7e-4185-4a1d-bfbf-916e41aa8b1d","Type":"ContainerDied","Data":"a167b2c6649d2b5148452cc0092397d177aad5513dcdb658eb2ffac152b1781d"}
Sep 16 05:12:11.057903 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.057887    2573 scope.go:117] "RemoveContainer" containerID="96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9"
Sep 16 05:12:11.058170 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:12:11.058151    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9\": container with ID starting with 96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9 not found: ID does not exist" containerID="96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9"
Sep 16 05:12:11.058243 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.058175    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9"} err="failed to get container status \"96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9\": rpc error: code = NotFound desc = could not find container \"96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9\": container with ID starting with 96170a4b0d3218e6a15645b44a4bf5317b46285144a164e178f30371357516b9 not found: ID does not exist"
Sep 16 05:12:11.058243 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.058191    2573 scope.go:117] "RemoveContainer" containerID="d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d"
Sep 16 05:12:11.066202 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.066186    2573 scope.go:117] "RemoveContainer" containerID="d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d"
Sep 16 05:12:11.066453 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:12:11.066437    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d\": container with ID starting with d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d not found: ID does not exist" containerID="d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d"
Sep 16 05:12:11.066543 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.066458    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d"} err="failed to get container status \"d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d\": rpc error: code = NotFound desc = could not find container \"d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d\": container with ID starting with d898fa18ba4d8875e903aabd0512a939c1839d8892e2aecc2732a48a06f2e34d not found: ID does not exist"
Sep 16 05:12:11.071974 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.071955    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"]
Sep 16 05:12:11.077828 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.077800    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-766a69c0-76f869779-ggght"]
Sep 16 05:12:11.085995 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.085971    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"]
Sep 16 05:12:11.090640 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.090620    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766a69c0-54c5798569-pm56d"]
Sep 16 05:12:11.162035 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.162009    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:11.836583 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:11.836503    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:12.558134 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:12.558096    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:12.787782 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:12.787746    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20485994-cd05-490e-a1f7-790ecc75109d" path="/var/lib/kubelet/pods/20485994-cd05-490e-a1f7-790ecc75109d/volumes"
Sep 16 05:12:12.788136 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:12.788123    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4303fb7e-4185-4a1d-bfbf-916e41aa8b1d" path="/var/lib/kubelet/pods/4303fb7e-4185-4a1d-bfbf-916e41aa8b1d/volumes"
Sep 16 05:12:13.235774 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:13.235742    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:13.771325 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:13.771283    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:13.910250 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:13.910217    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:14.591508 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:14.591481    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:15.261778 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:15.261737    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:18.015746 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:18.015718    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:18.841411 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:18.841373    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:23.852130 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.852102    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"]
Sep 16 05:12:23.852505 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.852444    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4303fb7e-4185-4a1d-bfbf-916e41aa8b1d" containerName="payload-processing"
Sep 16 05:12:23.852505 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.852456    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4303fb7e-4185-4a1d-bfbf-916e41aa8b1d" containerName="payload-processing"
Sep 16 05:12:23.852505 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.852473    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20485994-cd05-490e-a1f7-790ecc75109d" containerName="maas-api"
Sep 16 05:12:23.852505 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.852480    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="20485994-cd05-490e-a1f7-790ecc75109d" containerName="maas-api"
Sep 16 05:12:23.852710 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.852548    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="20485994-cd05-490e-a1f7-790ecc75109d" containerName="maas-api"
Sep 16 05:12:23.852710 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.852555    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4303fb7e-4185-4a1d-bfbf-916e41aa8b1d" containerName="payload-processing"
Sep 16 05:12:23.857149 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.857128    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:23.859814 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.859791    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-10ee79\"/\"kube-root-ca.crt\""
Sep 16 05:12:23.859925 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.859821    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-10ee79\"/\"openshift-service-ca.crt\""
Sep 16 05:12:23.859925 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.859842    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-10ee79\"/\"default-dockercfg-7dskn\""
Sep 16 05:12:23.859925 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.859791    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-10ee79\"/\"auth-test-model-10ee79-kserve-self-signed-certs\""
Sep 16 05:12:23.863838 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.863817    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"]
Sep 16 05:12:23.881399 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.881380    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:23.948817 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.948786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e59d152-ce8d-4e87-9f4e-31370462f72d-tls-certs\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:23.948930 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.948831    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-tmp-dir\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:23.948930 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.948907    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t565f\" (UniqueName: \"kubernetes.io/projected/7e59d152-ce8d-4e87-9f4e-31370462f72d-kube-api-access-t565f\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:23.949056 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.948983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-home\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:23.949056 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.949044    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-model-cache\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:23.949151 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:23.949073    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-dshm\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.050378 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.050354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-model-cache\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.050591 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.050571    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-dshm\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.050815 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.050798    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e59d152-ce8d-4e87-9f4e-31370462f72d-tls-certs\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.050971 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.050955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-tmp-dir\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.051087 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.051073    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t565f\" (UniqueName: \"kubernetes.io/projected/7e59d152-ce8d-4e87-9f4e-31370462f72d-kube-api-access-t565f\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.051210 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.051196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-home\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.051699 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.051676    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-home\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.052562 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.052543    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-tmp-dir\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.054251 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.054164    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-model-cache\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.054421 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.054400    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-dshm\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.057288 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.057249    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e59d152-ce8d-4e87-9f4e-31370462f72d-tls-certs\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.061294 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.061251    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t565f\" (UniqueName: \"kubernetes.io/projected/7e59d152-ce8d-4e87-9f4e-31370462f72d-kube-api-access-t565f\") pod \"auth-test-model-10ee79-kserve-7d96c94599-l2wlg\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.168312 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.168282    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:24.436932 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:24.436897    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"]
Sep 16 05:12:25.122838 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:25.122803    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg" event={"ID":"7e59d152-ce8d-4e87-9f4e-31370462f72d","Type":"ContainerStarted","Data":"8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d"}
Sep 16 05:12:25.123207 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:25.122849    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg" event={"ID":"7e59d152-ce8d-4e87-9f4e-31370462f72d","Type":"ContainerStarted","Data":"5e158a7e001bb4b32d293bccc7f04ee618c3d104e155442d06efe423f2720e85"}
Sep 16 05:12:25.140454 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:25.140408    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg" podStartSLOduration=2.140394155 podStartE2EDuration="2.140394155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:12:25.138129409 +0000 UTC m=+1408.804059004" watchObservedRunningTime="2026-09-16 05:12:25.140394155 +0000 UTC m=+1408.806323749"
Sep 16 05:12:34.169316 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:34.169283    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:34.169775 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:34.169329    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:34.181397 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:34.181376    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:35.184121 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:35.184088    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:12:42.084936 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:42.084897    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:12:42.368017 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:42.367936    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:55.624051 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:55.623942    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:12:55.763457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:12:55.763427    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:01.356495 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:01.356466    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:02.936954 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:02.936920    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:06.303941 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:06.303908    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:14.788386 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:14.788350    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:13:20.523435 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:20.523397    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:24.208946 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:24.208914    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:28.342019 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:28.341990    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:28.405657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:28.405628    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:37.243356 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:37.243323    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:42.210507 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:42.210472    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:42.311308 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:42.311274    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:42.419248 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:42.419219    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:43.668524 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:43.668489    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:43.763990 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:43.763953    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:43.864478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:43.864449    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:44.401883 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:44.401847    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:44.497498 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:44.497468    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:44.597042 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:44.597013    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:45.793464 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:45.793436    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:46.241752 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:46.241717    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:13:46.547583 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:46.547493    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:51.633183 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:51.633132    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:51.701325 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:51.701291    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:52.306907 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:52.306881    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:56.813956 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:56.813929    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:13:56.814982 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:56.814963    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:13:58.222516 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:58.222490    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:58.330199 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:58.330174    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:58.422744 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:58.422711    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:59.230847 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:59.230806    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:13:59.333659 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:13:59.333615    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:08.946530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:08.946496    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:09.146502 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:09.146467    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:17.298661 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:17.298631    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:22.304991 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:22.304954    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:26.990116 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:26.990079    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:14:30.079757 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:30.079720    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:34.753084 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:34.753054    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:35.227730 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:35.227697    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:36.023783 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:36.023753    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:36.093504 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:36.093471    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:36.250439 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:36.250411    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:36.350461 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:36.350378    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:36.763507 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:36.763481    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:38.241121 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:38.241088    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:38.548781 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:38.548702    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:39.307708 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:39.307671    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:39.729937 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:39.729902    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"]
Sep 16 05:14:39.755201 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:39.755172    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"]
Sep 16 05:14:39.768727 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:39.768700    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"]
Sep 16 05:14:39.826881 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:39.826853    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:40.249556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:40.249519    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:40.347263 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:40.347233    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:41.066242 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.066206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:41.830234 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.830195    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"]
Sep 16 05:14:41.833572 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.833549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.836671 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.836651    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-75559a-serving-cert\""
Sep 16 05:14:41.854383 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.854352    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"]
Sep 16 05:14:41.884962 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.884924    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-openshift-service-ca\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.885193 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.885087    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhjvh\" (UniqueName: \"kubernetes.io/projected/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-kube-api-access-qhjvh\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.885193 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.885147    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-metrics-certs\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.885333 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.885214    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-maas-api-tls\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.906115 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.906086    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:41.986242 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.986205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-openshift-service-ca\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.986242 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.986242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhjvh\" (UniqueName: \"kubernetes.io/projected/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-kube-api-access-qhjvh\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.986516 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.986369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-metrics-certs\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.986516 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.986415    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-maas-api-tls\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.987003 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.986977    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-openshift-service-ca\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.988958 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.988935    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-metrics-certs\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.988958 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.988949    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-maas-api-tls\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:41.993746 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:41.993723    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhjvh\" (UniqueName: \"kubernetes.io/projected/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-kube-api-access-qhjvh\") pod \"maas-api-e2e-api-a-75559a-6dddb74f64-fbv54\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:42.055450 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.055420    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:42.143513 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.143473    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:42.156739 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.156716    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:42.376721 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.376690    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:42.424286 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.424150    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"]
Sep 16 05:14:42.426845 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:14:42.426814    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b111d3_dfa0_452e_a555_8eb7d8bfa733.slice/crio-1de02f5209b914baa9c0bdd43c9529e8d44ab6abb566a6a2700e8dd1d55d049a WatchSource:0}: Error finding container 1de02f5209b914baa9c0bdd43c9529e8d44ab6abb566a6a2700e8dd1d55d049a: Status 404 returned error can't find the container with id 1de02f5209b914baa9c0bdd43c9529e8d44ab6abb566a6a2700e8dd1d55d049a
Sep 16 05:14:42.428251 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.428231    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 05:14:42.476177 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.476139    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:42.476528 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.476511    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"]
Sep 16 05:14:42.479684 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.479668    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:14:42.482308 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.482284    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-75559a-dockercfg-l762d\""
Sep 16 05:14:42.482635 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.482618    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-75559a\""
Sep 16 05:14:42.492051 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.492021    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"]
Sep 16 05:14:42.593065 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.593023    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxtq\" (UniqueName: \"kubernetes.io/projected/56b820c8-7119-4293-b296-c923d06f2d1b-kube-api-access-vxxtq\") pod \"payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d\" (UID: \"56b820c8-7119-4293-b296-c923d06f2d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:14:42.593317 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.593100    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b820c8-7119-4293-b296-c923d06f2d1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d\" (UID: \"56b820c8-7119-4293-b296-c923d06f2d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:14:42.693981 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.693880    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxtq\" (UniqueName: \"kubernetes.io/projected/56b820c8-7119-4293-b296-c923d06f2d1b-kube-api-access-vxxtq\") pod \"payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d\" (UID: \"56b820c8-7119-4293-b296-c923d06f2d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:14:42.693981 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.693931    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b820c8-7119-4293-b296-c923d06f2d1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d\" (UID: \"56b820c8-7119-4293-b296-c923d06f2d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:14:42.694656 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.694636    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b820c8-7119-4293-b296-c923d06f2d1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d\" (UID: \"56b820c8-7119-4293-b296-c923d06f2d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:14:42.703020 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.702989    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxtq\" (UniqueName: \"kubernetes.io/projected/56b820c8-7119-4293-b296-c923d06f2d1b-kube-api-access-vxxtq\") pod \"payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d\" (UID: \"56b820c8-7119-4293-b296-c923d06f2d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:14:42.791217 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.791183    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:14:42.807620 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.807565    2573 generic.go:356] "Generic (PLEG): container finished" podID="6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" containerID="8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f" exitCode=137
Sep 16 05:14:42.807727 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.807645    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg" event={"ID":"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003","Type":"ContainerDied","Data":"8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f"}
Sep 16 05:14:42.808039 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.808013    2573 scope.go:117] "RemoveContainer" containerID="8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f"
Sep 16 05:14:42.808951 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.808931    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54" event={"ID":"b6b111d3-dfa0-452e-a555-8eb7d8bfa733","Type":"ContainerStarted","Data":"1de02f5209b914baa9c0bdd43c9529e8d44ab6abb566a6a2700e8dd1d55d049a"}
Sep 16 05:14:42.948924 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:42.948843    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:43.081072 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.081049    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"]
Sep 16 05:14:43.083751 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:14:43.083710    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b820c8_7119_4293_b296_c923d06f2d1b.slice/crio-5fddced065fde14314981de9efbb1ee56ac1f9e17cb0d5449cccbabd65dfd2d3 WatchSource:0}: Error finding container 5fddced065fde14314981de9efbb1ee56ac1f9e17cb0d5449cccbabd65dfd2d3: Status 404 returned error can't find the container with id 5fddced065fde14314981de9efbb1ee56ac1f9e17cb0d5449cccbabd65dfd2d3
Sep 16 05:14:43.816282 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.816240    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg" event={"ID":"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003","Type":"ContainerStarted","Data":"08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735"}
Sep 16 05:14:43.816510 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.816491    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 05:14:43.817219 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.817189    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:43.818055 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.818033    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54" event={"ID":"b6b111d3-dfa0-452e-a555-8eb7d8bfa733","Type":"ContainerStarted","Data":"a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a"}
Sep 16 05:14:43.818163 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.818089    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:43.822723 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.822384    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d" event={"ID":"56b820c8-7119-4293-b296-c923d06f2d1b","Type":"ContainerStarted","Data":"c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312"}
Sep 16 05:14:43.822723 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.822440    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d" event={"ID":"56b820c8-7119-4293-b296-c923d06f2d1b","Type":"ContainerStarted","Data":"5fddced065fde14314981de9efbb1ee56ac1f9e17cb0d5449cccbabd65dfd2d3"}
Sep 16 05:14:43.822723 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.822649    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:14:43.870803 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.870744    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d" podStartSLOduration=1.870724233 podStartE2EDuration="1.870724233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:14:43.870498967 +0000 UTC m=+1547.536428581" watchObservedRunningTime="2026-09-16 05:14:43.870724233 +0000 UTC m=+1547.536653827"
Sep 16 05:14:43.892229 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:43.892164    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54" podStartSLOduration=2.14955691 podStartE2EDuration="2.892146345s" totalImagesPullingTime="742.589435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:14:43.892026179 +0000 UTC m=+1547.557955825" watchObservedRunningTime="2026-09-16 05:14:43.892146345 +0000 UTC m=+1547.558075938"
Sep 16 05:14:44.709325 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:44.709293    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:45.504853 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:45.504821    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:46.435303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:46.435270    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:47.248951 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:47.248919    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:49.833887 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:49.833858    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:14:52.887438 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:52.887404    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"]
Sep 16 05:14:52.887922 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:52.887652    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c" podUID="b19c8295-d377-4625-b588-942f7214ab31" containerName="maas-api" containerID="cri-o://d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3" gracePeriod=30
Sep 16 05:14:52.917656 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:52.917621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:53.018525 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.018489    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:53.111496 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.111465    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:53.165887 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.165790    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:53.347919 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.347893    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:14:53.453953 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.453862    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:53.500330 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.500284    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t84rx\" (UniqueName: \"kubernetes.io/projected/b19c8295-d377-4625-b588-942f7214ab31-kube-api-access-t84rx\") pod \"b19c8295-d377-4625-b588-942f7214ab31\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") "
Sep 16 05:14:53.500541 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.500405    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19c8295-d377-4625-b588-942f7214ab31-openshift-service-ca\") pod \"b19c8295-d377-4625-b588-942f7214ab31\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") "
Sep 16 05:14:53.500541 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.500502    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-maas-api-tls\") pod \"b19c8295-d377-4625-b588-942f7214ab31\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") "
Sep 16 05:14:53.500679 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.500547    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-metrics-certs\") pod \"b19c8295-d377-4625-b588-942f7214ab31\" (UID: \"b19c8295-d377-4625-b588-942f7214ab31\") "
Sep 16 05:14:53.500792 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.500762    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19c8295-d377-4625-b588-942f7214ab31-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b19c8295-d377-4625-b588-942f7214ab31" (UID: "b19c8295-d377-4625-b588-942f7214ab31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:14:53.500898 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.500881    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19c8295-d377-4625-b588-942f7214ab31-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:53.502506 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.502472    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19c8295-d377-4625-b588-942f7214ab31-kube-api-access-t84rx" (OuterVolumeSpecName: "kube-api-access-t84rx") pod "b19c8295-d377-4625-b588-942f7214ab31" (UID: "b19c8295-d377-4625-b588-942f7214ab31"). InnerVolumeSpecName "kube-api-access-t84rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:14:53.502636 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.502551    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b19c8295-d377-4625-b588-942f7214ab31" (UID: "b19c8295-d377-4625-b588-942f7214ab31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:14:53.502636 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.502575    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b19c8295-d377-4625-b588-942f7214ab31" (UID: "b19c8295-d377-4625-b588-942f7214ab31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:14:53.553392 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.553361    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:53.601739 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.601704    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:53.601739 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.601733    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19c8295-d377-4625-b588-942f7214ab31-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:53.601739 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.601743    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t84rx\" (UniqueName: \"kubernetes.io/projected/b19c8295-d377-4625-b588-942f7214ab31-kube-api-access-t84rx\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:53.884931 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.884896    2573 generic.go:356] "Generic (PLEG): container finished" podID="b19c8295-d377-4625-b588-942f7214ab31" containerID="d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3" exitCode=0
Sep 16 05:14:53.885132 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.884938    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"
Sep 16 05:14:53.885132 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.884980    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c" event={"ID":"b19c8295-d377-4625-b588-942f7214ab31","Type":"ContainerDied","Data":"d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3"}
Sep 16 05:14:53.885132 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.885035    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c" event={"ID":"b19c8295-d377-4625-b588-942f7214ab31","Type":"ContainerDied","Data":"2daefd82430b52bf5e9332887581992add4a09f35493c1e7bf5115c410ca85cd"}
Sep 16 05:14:53.885132 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.885067    2573 scope.go:117] "RemoveContainer" containerID="d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3"
Sep 16 05:14:53.895154 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.894990    2573 scope.go:117] "RemoveContainer" containerID="d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3"
Sep 16 05:14:53.895392 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:14:53.895298    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3\": container with ID starting with d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3 not found: ID does not exist" containerID="d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3"
Sep 16 05:14:53.895392 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.895326    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3"} err="failed to get container status \"d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3\": rpc error: code = NotFound desc = could not find container \"d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3\": container with ID starting with d1974b23a9c3efc5e819bcca1f5eecfdc3037183d652d03928c8824c7d227ae3 not found: ID does not exist"
Sep 16 05:14:53.907025 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.906992    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"]
Sep 16 05:14:53.911140 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:53.911100    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee7a60-7878648fb8-t4j2c"]
Sep 16 05:14:54.306249 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.306157    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:54.749851 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.749806    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl" podUID="b6b6a982-d4fa-476d-87c5-45972dadf1c4" containerName="main" containerID="cri-o://d273b49f24feb546f721df546dc6ba06a6920ea9fec87283bb3f0dd0114ceb1c" gracePeriod=45
Sep 16 05:14:54.774937 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.774891    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v" podUID="8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" containerName="main" containerID="cri-o://a55a19a50e2e4db2629396f39aac641aad848d5088a264772c1e1de465188935" gracePeriod=45
Sep 16 05:14:54.787142 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.787107    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b19c8295-d377-4625-b588-942f7214ab31" path="/var/lib/kubelet/pods/b19c8295-d377-4625-b588-942f7214ab31/volumes"
Sep 16 05:14:54.789089 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.789059    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b" podUID="b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" containerName="main" containerID="cri-o://879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c" gracePeriod=45
Sep 16 05:14:54.830295 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.830261    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 05:14:54.830432 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.830338    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:14:54.892216 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.892174    2573 generic.go:356] "Generic (PLEG): container finished" podID="8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" containerID="a55a19a50e2e4db2629396f39aac641aad848d5088a264772c1e1de465188935" exitCode=0
Sep 16 05:14:54.892378 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.892265    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v" event={"ID":"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5","Type":"ContainerDied","Data":"a55a19a50e2e4db2629396f39aac641aad848d5088a264772c1e1de465188935"}
Sep 16 05:14:54.894108 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.894085    2573 generic.go:356] "Generic (PLEG): container finished" podID="b6b6a982-d4fa-476d-87c5-45972dadf1c4" containerID="d273b49f24feb546f721df546dc6ba06a6920ea9fec87283bb3f0dd0114ceb1c" exitCode=0
Sep 16 05:14:54.894247 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.894161    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl" event={"ID":"b6b6a982-d4fa-476d-87c5-45972dadf1c4","Type":"ContainerDied","Data":"d273b49f24feb546f721df546dc6ba06a6920ea9fec87283bb3f0dd0114ceb1c"}
Sep 16 05:14:54.962382 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:54.962330    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:55.121062 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.121027    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:14:55.125019 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.124993    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:14:55.130364 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.130339    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:14:55.216877 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.216845    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tls-certs\") pod \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") "
Sep 16 05:14:55.217045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.216893    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tls-certs\") pod \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") "
Sep 16 05:14:55.217045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.216915    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-home\") pod \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") "
Sep 16 05:14:55.217045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.216937    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tmp-dir\") pod \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") "
Sep 16 05:14:55.217045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.216970    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk6qz\" (UniqueName: \"kubernetes.io/projected/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-kube-api-access-vk6qz\") pod \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") "
Sep 16 05:14:55.217045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217034    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tls-certs\") pod \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") "
Sep 16 05:14:55.217303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217057    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-model-cache\") pod \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") "
Sep 16 05:14:55.217303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217085    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-dshm\") pod \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") "
Sep 16 05:14:55.217303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217112    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-dshm\") pod \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") "
Sep 16 05:14:55.217303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217150    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tmp-dir\") pod \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") "
Sep 16 05:14:55.217303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217173    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-home\") pod \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") "
Sep 16 05:14:55.217303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217196    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-dshm\") pod \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") "
Sep 16 05:14:55.217303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217214    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-home" (OuterVolumeSpecName: "home") pod "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" (UID: "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.217303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217231    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-model-cache\") pod \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\" (UID: \"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5\") "
Sep 16 05:14:55.217303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217256    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-model-cache\") pod \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") "
Sep 16 05:14:55.217777 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217315    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tmp-dir\") pod \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") "
Sep 16 05:14:55.217777 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217341    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2rv2\" (UniqueName: \"kubernetes.io/projected/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-kube-api-access-x2rv2\") pod \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\" (UID: \"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde\") "
Sep 16 05:14:55.217777 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217368    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-home\") pod \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") "
Sep 16 05:14:55.217777 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217392    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24w5s\" (UniqueName: \"kubernetes.io/projected/b6b6a982-d4fa-476d-87c5-45972dadf1c4-kube-api-access-24w5s\") pod \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\" (UID: \"b6b6a982-d4fa-476d-87c5-45972dadf1c4\") "
Sep 16 05:14:55.217777 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217414    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" (UID: "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.217777 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217505    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" (UID: "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.218147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217821    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.218147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217844    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.218147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.217859    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.218147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.218101    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6b6a982-d4fa-476d-87c5-45972dadf1c4" (UID: "b6b6a982-d4fa-476d-87c5-45972dadf1c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.218351 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.218269    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" (UID: "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.218504 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.218478    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" (UID: "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.218772 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.218710    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-home" (OuterVolumeSpecName: "home") pod "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" (UID: "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.218940 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.218897    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-home" (OuterVolumeSpecName: "home") pod "b6b6a982-d4fa-476d-87c5-45972dadf1c4" (UID: "b6b6a982-d4fa-476d-87c5-45972dadf1c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.219006 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.218942    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6b6a982-d4fa-476d-87c5-45972dadf1c4" (UID: "b6b6a982-d4fa-476d-87c5-45972dadf1c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.220382 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.220346    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6b6a982-d4fa-476d-87c5-45972dadf1c4" (UID: "b6b6a982-d4fa-476d-87c5-45972dadf1c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:14:55.220514 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.220459    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b6a982-d4fa-476d-87c5-45972dadf1c4-kube-api-access-24w5s" (OuterVolumeSpecName: "kube-api-access-24w5s") pod "b6b6a982-d4fa-476d-87c5-45972dadf1c4" (UID: "b6b6a982-d4fa-476d-87c5-45972dadf1c4"). InnerVolumeSpecName "kube-api-access-24w5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:14:55.220514 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.220484    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-kube-api-access-vk6qz" (OuterVolumeSpecName: "kube-api-access-vk6qz") pod "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" (UID: "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5"). InnerVolumeSpecName "kube-api-access-vk6qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:14:55.220514 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.220501    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-dshm" (OuterVolumeSpecName: "dshm") pod "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" (UID: "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.220757 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.220526    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" (UID: "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:14:55.220821 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.220788    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-dshm" (OuterVolumeSpecName: "dshm") pod "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" (UID: "8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.221559 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.221534    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-dshm" (OuterVolumeSpecName: "dshm") pod "b6b6a982-d4fa-476d-87c5-45972dadf1c4" (UID: "b6b6a982-d4fa-476d-87c5-45972dadf1c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:14:55.221923 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.221903    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" (UID: "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:14:55.222037 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.222021    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-kube-api-access-x2rv2" (OuterVolumeSpecName: "kube-api-access-x2rv2") pod "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" (UID: "b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde"). InnerVolumeSpecName "kube-api-access-x2rv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:14:55.319363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319270    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319300    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319309    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319319    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319327    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319335    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2rv2\" (UniqueName: \"kubernetes.io/projected/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-kube-api-access-x2rv2\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319344    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319352    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24w5s\" (UniqueName: \"kubernetes.io/projected/b6b6a982-d4fa-476d-87c5-45972dadf1c4-kube-api-access-24w5s\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319360    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319363 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319368    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319830 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319376    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk6qz\" (UniqueName: \"kubernetes.io/projected/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-kube-api-access-vk6qz\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319830 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319384    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b6a982-d4fa-476d-87c5-45972dadf1c4-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319830 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319392    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6b6a982-d4fa-476d-87c5-45972dadf1c4-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319830 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319398    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.319830 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.319407    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:14:55.874969 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.874930    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:55.902202 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.902169    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"
Sep 16 05:14:55.902380 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.902201    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v" event={"ID":"8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5","Type":"ContainerDied","Data":"55aba706ab35438cf309d061e06f6d7d60a5fe532dd603a2c06bfd7a0e26fa80"}
Sep 16 05:14:55.902380 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.902251    2573 scope.go:117] "RemoveContainer" containerID="a55a19a50e2e4db2629396f39aac641aad848d5088a264772c1e1de465188935"
Sep 16 05:14:55.903675 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.903591    2573 generic.go:356] "Generic (PLEG): container finished" podID="b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" containerID="879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c" exitCode=0
Sep 16 05:14:55.903675 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.903667    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"
Sep 16 05:14:55.903826 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.903682    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b" event={"ID":"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde","Type":"ContainerDied","Data":"879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c"}
Sep 16 05:14:55.903826 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.903721    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b" event={"ID":"b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde","Type":"ContainerDied","Data":"5b43014ba6ae67217b8bf022156a569dd8e423f078394f239f65fa4a5aa157c2"}
Sep 16 05:14:55.905343 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.905309    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl" event={"ID":"b6b6a982-d4fa-476d-87c5-45972dadf1c4","Type":"ContainerDied","Data":"f9cbb9e55eceff8166bfea2416b9935a9d448399e6b942393db2f54b70add788"}
Sep 16 05:14:55.905469 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.905408    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"
Sep 16 05:14:55.911803 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.911771    2573 scope.go:117] "RemoveContainer" containerID="879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c"
Sep 16 05:14:55.922513 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.922493    2573 scope.go:117] "RemoveContainer" containerID="879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c"
Sep 16 05:14:55.922841 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:14:55.922817    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c\": container with ID starting with 879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c not found: ID does not exist" containerID="879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c"
Sep 16 05:14:55.922899 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.922849    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c"} err="failed to get container status \"879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c\": rpc error: code = NotFound desc = could not find container \"879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c\": container with ID starting with 879ca1726673f27a5a17fcb7672c12f6340d49e382c0b6ad9ad6c7a265cc616c not found: ID does not exist"
Sep 16 05:14:55.922899 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.922865    2573 scope.go:117] "RemoveContainer" containerID="d273b49f24feb546f721df546dc6ba06a6920ea9fec87283bb3f0dd0114ceb1c"
Sep 16 05:14:55.931844 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.931781    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"]
Sep 16 05:14:55.938701 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.938677    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/distinct-w0-ee7a60-kserve-68dff5c9ff-ttskl"]
Sep 16 05:14:55.952331 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.952296    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"]
Sep 16 05:14:55.957117 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.957094    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/premium-w0-ee7a60-kserve-6df8cb448-t8d4v"]
Sep 16 05:14:55.969635 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.969590    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"]
Sep 16 05:14:55.973747 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:55.973726    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/simulator-w0-ee7a60-kserve-97bbc548f-w4k2b"]
Sep 16 05:14:56.762400 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:56.762366    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:56.786808 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:56.786775    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" path="/var/lib/kubelet/pods/8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5/volumes"
Sep 16 05:14:56.787197 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:56.787185    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6b6a982-d4fa-476d-87c5-45972dadf1c4" path="/var/lib/kubelet/pods/b6b6a982-d4fa-476d-87c5-45972dadf1c4/volumes"
Sep 16 05:14:56.787538 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:56.787527    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" path="/var/lib/kubelet/pods/b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde/volumes"
Sep 16 05:14:57.569666 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:57.569632    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:58.204550 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:58.204504    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:58.832766 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:58.832732    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:59.455170 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:59.455138    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:59.741653 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:59.741556    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:14:59.842484 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:14:59.842451    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:00.026137 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.026056    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:00.133238 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.133193    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:00.176970 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.176938    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825595-l9tpn"]
Sep 16 05:15:00.177314 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177301    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" containerName="main"
Sep 16 05:15:00.177360 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177315    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" containerName="main"
Sep 16 05:15:00.177360 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177328    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6b6a982-d4fa-476d-87c5-45972dadf1c4" containerName="main"
Sep 16 05:15:00.177360 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177334    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6b6a982-d4fa-476d-87c5-45972dadf1c4" containerName="main"
Sep 16 05:15:00.177360 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177353    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" containerName="main"
Sep 16 05:15:00.177360 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177358    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" containerName="main"
Sep 16 05:15:00.177509 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177366    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b19c8295-d377-4625-b588-942f7214ab31" containerName="maas-api"
Sep 16 05:15:00.177509 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177371    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b19c8295-d377-4625-b588-942f7214ab31" containerName="maas-api"
Sep 16 05:15:00.177509 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177427    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b19c8295-d377-4625-b588-942f7214ab31" containerName="maas-api"
Sep 16 05:15:00.177509 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177436    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6b6a982-d4fa-476d-87c5-45972dadf1c4" containerName="main"
Sep 16 05:15:00.177509 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177442    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c8b0ae-4ef6-4f77-abd8-0ffb114e6fde" containerName="main"
Sep 16 05:15:00.177509 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.177449    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a58a94b-fe7b-4a2b-8176-64d2e1a4eab5" containerName="main"
Sep 16 05:15:00.180419 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.180395    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825595-l9tpn"
Sep 16 05:15:00.182444 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.182422    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sm548\""
Sep 16 05:15:00.191276 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.191240    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825595-l9tpn"
Sep 16 05:15:00.191399 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.191376    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825595-l9tpn"]
Sep 16 05:15:00.233939 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.233895    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:00.476369 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.476320    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825595-l9tpn"]
Sep 16 05:15:00.479378 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:15:00.479352    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3964d8d_8453_4d80_9dfc_d66d2127aeba.slice/crio-c4a16db3be6ffd08923ade73ddb4904c5975301cda442e34d7a881de928e9dec WatchSource:0}: Error finding container c4a16db3be6ffd08923ade73ddb4904c5975301cda442e34d7a881de928e9dec: Status 404 returned error can't find the container with id c4a16db3be6ffd08923ade73ddb4904c5975301cda442e34d7a881de928e9dec
Sep 16 05:15:00.497403 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.497376    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:00.814045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.813964    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:00.913583 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.913552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:00.936487 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:00.936451    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825595-l9tpn" event={"ID":"d3964d8d-8453-4d80-9dfc-d66d2127aeba","Type":"ContainerStarted","Data":"c4a16db3be6ffd08923ade73ddb4904c5975301cda442e34d7a881de928e9dec"}
Sep 16 05:15:01.010874 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:01.010736    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:01.874032 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:01.873982    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:02.835983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:02.835950    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:03.474315 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:03.474289    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:03.603226 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:03.603194    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:03.708532 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:03.708499    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:03.953888 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:03.953855    2573 generic.go:356] "Generic (PLEG): container finished" podID="d3964d8d-8453-4d80-9dfc-d66d2127aeba" containerID="5be62c08b1dfdddd41b79446a09cfa5ad6fe35a99588bcb1be6370c9e59bd887" exitCode=0
Sep 16 05:15:03.954265 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:03.953941    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825595-l9tpn" event={"ID":"d3964d8d-8453-4d80-9dfc-d66d2127aeba","Type":"ContainerDied","Data":"5be62c08b1dfdddd41b79446a09cfa5ad6fe35a99588bcb1be6370c9e59bd887"}
Sep 16 05:15:04.671790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:04.671760    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:05.095921 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:05.095898    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825595-l9tpn"
Sep 16 05:15:05.585935 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:05.585903    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:05.965186 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:05.965149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825595-l9tpn" event={"ID":"d3964d8d-8453-4d80-9dfc-d66d2127aeba","Type":"ContainerDied","Data":"c4a16db3be6ffd08923ade73ddb4904c5975301cda442e34d7a881de928e9dec"}
Sep 16 05:15:05.965381 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:05.965201    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a16db3be6ffd08923ade73ddb4904c5975301cda442e34d7a881de928e9dec"
Sep 16 05:15:05.965381 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:05.965162    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825595-l9tpn"
Sep 16 05:15:06.629753 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:06.629724    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:07.539572 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:07.539536    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:08.159962 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:08.159928    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:08.862357 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:08.862322    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"]
Sep 16 05:15:08.862722 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:08.862708    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3964d8d-8453-4d80-9dfc-d66d2127aeba" containerName="cleanup"
Sep 16 05:15:08.862767 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:08.862724    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3964d8d-8453-4d80-9dfc-d66d2127aeba" containerName="cleanup"
Sep 16 05:15:08.862817 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:08.862793    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3964d8d-8453-4d80-9dfc-d66d2127aeba" containerName="cleanup"
Sep 16 05:15:08.865804 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:08.865782    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:08.867776 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:08.867752    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-113278-serving-cert\""
Sep 16 05:15:08.875836 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:08.875804    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"]
Sep 16 05:15:08.918522 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:08.918484    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:09.020160 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.020117    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:09.051045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.051010    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75v4\" (UniqueName: \"kubernetes.io/projected/7132a6d0-8c3f-4010-941b-085279fd7a97-kube-api-access-f75v4\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.051210 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.051115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7132a6d0-8c3f-4010-941b-085279fd7a97-openshift-service-ca\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.051210 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.051157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-maas-api-tls\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.051210 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.051194    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-metrics-certs\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.078887 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.078844    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:09.152204 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.152170    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7132a6d0-8c3f-4010-941b-085279fd7a97-openshift-service-ca\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.152393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.152217    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-maas-api-tls\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.152393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.152243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-metrics-certs\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.152393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.152300    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75v4\" (UniqueName: \"kubernetes.io/projected/7132a6d0-8c3f-4010-941b-085279fd7a97-kube-api-access-f75v4\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.152964 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.152936    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7132a6d0-8c3f-4010-941b-085279fd7a97-openshift-service-ca\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.154801 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.154780    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-metrics-certs\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.154948 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.154926    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-maas-api-tls\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.160723 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.160697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75v4\" (UniqueName: \"kubernetes.io/projected/7132a6d0-8c3f-4010-941b-085279fd7a97-kube-api-access-f75v4\") pod \"maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.179562 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.179537    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:09.346779 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.346746    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:09.444810 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.444738    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:09.453347 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.453318    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"]
Sep 16 05:15:09.458034 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.458009    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:15:09.459859 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.459838    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"]
Sep 16 05:15:09.460325 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.460293    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-113278-dockercfg-bg5lq\""
Sep 16 05:15:09.460373 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.460328    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-113278\""
Sep 16 05:15:09.462928 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:15:09.462900    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7132a6d0_8c3f_4010_941b_085279fd7a97.slice/crio-a9197966936b74497e8d8c77fbadab862f65650e961594fcfaa374e9651b59f8 WatchSource:0}: Error finding container a9197966936b74497e8d8c77fbadab862f65650e961594fcfaa374e9651b59f8: Status 404 returned error can't find the container with id a9197966936b74497e8d8c77fbadab862f65650e961594fcfaa374e9651b59f8
Sep 16 05:15:09.466483 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.466462    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"]
Sep 16 05:15:09.499761 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.499726    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"]
Sep 16 05:15:09.503200 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.503181    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:15:09.516809 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.516772    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"]
Sep 16 05:15:09.556063 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.556027    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9cr\" (UniqueName: \"kubernetes.io/projected/238c97f5-b96a-4661-a772-158b0cbf54ad-kube-api-access-9r9cr\") pod \"payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj\" (UID: \"238c97f5-b96a-4661-a772-158b0cbf54ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:15:09.556224 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.556072    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238c97f5-b96a-4661-a772-158b0cbf54ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj\" (UID: \"238c97f5-b96a-4661-a772-158b0cbf54ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:15:09.657751 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.657704    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9f8959-2563-48fb-962b-a882b9aa6f5c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-113278-78788d5969-v9rxb\" (UID: \"6e9f8959-2563-48fb-962b-a882b9aa6f5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:15:09.657751 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.657761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9cr\" (UniqueName: \"kubernetes.io/projected/238c97f5-b96a-4661-a772-158b0cbf54ad-kube-api-access-9r9cr\") pod \"payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj\" (UID: \"238c97f5-b96a-4661-a772-158b0cbf54ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:15:09.658033 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.657781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238c97f5-b96a-4661-a772-158b0cbf54ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj\" (UID: \"238c97f5-b96a-4661-a772-158b0cbf54ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:15:09.658033 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.657848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbftr\" (UniqueName: \"kubernetes.io/projected/6e9f8959-2563-48fb-962b-a882b9aa6f5c-kube-api-access-xbftr\") pod \"payload-processing-e2e-api-b-113278-78788d5969-v9rxb\" (UID: \"6e9f8959-2563-48fb-962b-a882b9aa6f5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:15:09.658468 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.658445    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238c97f5-b96a-4661-a772-158b0cbf54ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj\" (UID: \"238c97f5-b96a-4661-a772-158b0cbf54ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:15:09.668216 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.668185    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9cr\" (UniqueName: \"kubernetes.io/projected/238c97f5-b96a-4661-a772-158b0cbf54ad-kube-api-access-9r9cr\") pod \"payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj\" (UID: \"238c97f5-b96a-4661-a772-158b0cbf54ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:15:09.758686 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.758551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9f8959-2563-48fb-962b-a882b9aa6f5c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-113278-78788d5969-v9rxb\" (UID: \"6e9f8959-2563-48fb-962b-a882b9aa6f5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:15:09.758686 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.758663    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbftr\" (UniqueName: \"kubernetes.io/projected/6e9f8959-2563-48fb-962b-a882b9aa6f5c-kube-api-access-xbftr\") pod \"payload-processing-e2e-api-b-113278-78788d5969-v9rxb\" (UID: \"6e9f8959-2563-48fb-962b-a882b9aa6f5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:15:09.759367 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.759338    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9f8959-2563-48fb-962b-a882b9aa6f5c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-113278-78788d5969-v9rxb\" (UID: \"6e9f8959-2563-48fb-962b-a882b9aa6f5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:15:09.768519 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.768482    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbftr\" (UniqueName: \"kubernetes.io/projected/6e9f8959-2563-48fb-962b-a882b9aa6f5c-kube-api-access-xbftr\") pod \"payload-processing-e2e-api-b-113278-78788d5969-v9rxb\" (UID: \"6e9f8959-2563-48fb-962b-a882b9aa6f5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:15:09.778754 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.778727    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:15:09.814965 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.814922    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:15:09.984719 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:09.984683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8" event={"ID":"7132a6d0-8c3f-4010-941b-085279fd7a97","Type":"ContainerStarted","Data":"a9197966936b74497e8d8c77fbadab862f65650e961594fcfaa374e9651b59f8"}
Sep 16 05:15:10.072214 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:10.072182    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"]
Sep 16 05:15:10.073068 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:15:10.073039    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238c97f5_b96a_4661_a772_158b0cbf54ad.slice/crio-063889493af6d2d81f26bbc826238aac0c4393a49749816a1b3767f08c4c449c WatchSource:0}: Error finding container 063889493af6d2d81f26bbc826238aac0c4393a49749816a1b3767f08c4c449c: Status 404 returned error can't find the container with id 063889493af6d2d81f26bbc826238aac0c4393a49749816a1b3767f08c4c449c
Sep 16 05:15:10.105965 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:10.105941    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"]
Sep 16 05:15:10.108771 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:15:10.108744    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9f8959_2563_48fb_962b_a882b9aa6f5c.slice/crio-a12b219ccd6064ea8d5f57793adfe1b870fe3fb56280559c8dc33ec24b47c92d WatchSource:0}: Error finding container a12b219ccd6064ea8d5f57793adfe1b870fe3fb56280559c8dc33ec24b47c92d: Status 404 returned error can't find the container with id a12b219ccd6064ea8d5f57793adfe1b870fe3fb56280559c8dc33ec24b47c92d
Sep 16 05:15:10.992887 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:10.992838    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8" event={"ID":"7132a6d0-8c3f-4010-941b-085279fd7a97","Type":"ContainerStarted","Data":"99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1"}
Sep 16 05:15:10.993343 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:10.992914    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:10.994515 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:10.994486    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj" event={"ID":"238c97f5-b96a-4661-a772-158b0cbf54ad","Type":"ContainerStarted","Data":"4afbc01dde6d4eb5648b09722206d35e6ae120876ccafba9735f00cfa3760298"}
Sep 16 05:15:10.994657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:10.994537    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj" event={"ID":"238c97f5-b96a-4661-a772-158b0cbf54ad","Type":"ContainerStarted","Data":"063889493af6d2d81f26bbc826238aac0c4393a49749816a1b3767f08c4c449c"}
Sep 16 05:15:10.994657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:10.994586    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:15:10.996284 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:10.996262    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb" event={"ID":"6e9f8959-2563-48fb-962b-a882b9aa6f5c","Type":"ContainerStarted","Data":"a85e08592445dc7493b62e4f01821699d3925c48e5021b6ec870dacffb40ca09"}
Sep 16 05:15:10.996402 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:10.996297    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb" event={"ID":"6e9f8959-2563-48fb-962b-a882b9aa6f5c","Type":"ContainerStarted","Data":"a12b219ccd6064ea8d5f57793adfe1b870fe3fb56280559c8dc33ec24b47c92d"}
Sep 16 05:15:10.996402 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:10.996325    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:15:11.015201 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:11.015155    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8" podStartSLOduration=2.553887225 podStartE2EDuration="3.0151391s" totalImagesPullingTime="461.251875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:15:11.011340315 +0000 UTC m=+1574.677269908" watchObservedRunningTime="2026-09-16 05:15:11.0151391 +0000 UTC m=+1574.681068718"
Sep 16 05:15:11.033392 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:11.033342    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb" podStartSLOduration=2.033328521 podStartE2EDuration="2.033328521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:15:11.031021377 +0000 UTC m=+1574.696950998" watchObservedRunningTime="2026-09-16 05:15:11.033328521 +0000 UTC m=+1574.699258113"
Sep 16 05:15:11.049431 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:11.049376    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj" podStartSLOduration=2.049361302 podStartE2EDuration="2.049361302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:15:11.047943168 +0000 UTC m=+1574.713872782" watchObservedRunningTime="2026-09-16 05:15:11.049361302 +0000 UTC m=+1574.715290946"
Sep 16 05:15:11.662536 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:11.662472    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:11.765240 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:11.765192    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:11.902477 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:11.902440    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:12.691399 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:12.691362    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:15:14.123864 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:14.123818    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:17.007137 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:17.007109    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:15:17.618336 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:17.618306    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:22.003651 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:22.003614    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:15:22.004075 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:22.003788    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:15:23.363300 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:23.363259    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:28.196960 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:28.196924    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:29.743161 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:29.743129    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:31.031541 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:31.031510    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:31.931240 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:31.931204    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:32.618208 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:32.618177    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:32.983411 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:32.983379    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:15:33.244826 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:33.244748    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:33.609179 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:33.609098    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:33.899186 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:33.899152    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:34.634529 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:34.634495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:35.431980 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:35.431952    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:36.140398 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:36.140372    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:37.205563 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:37.205519    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:37.848166 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:37.848129    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:38.559385 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:38.559358    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:39.449721 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:39.449670    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:39.978348 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:39.978320    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:40.657450 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:40.657418    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:41.551031 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:41.550996    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:42.343803 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:42.343770    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:46.624429 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:46.624396    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:47.942279 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:47.942249    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:52.189275 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.189196    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"]
Sep 16 05:15:52.189745 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.189438    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54" podUID="b6b111d3-dfa0-452e-a555-8eb7d8bfa733" containerName="maas-api" containerID="cri-o://a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a" gracePeriod=30
Sep 16 05:15:52.233829 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.233797    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:52.281083 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.281049    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"]
Sep 16 05:15:52.281336 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.281310    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d" podUID="56b820c8-7119-4293-b296-c923d06f2d1b" containerName="payload-pre-processing" containerID="cri-o://c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312" gracePeriod=30
Sep 16 05:15:52.409788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.409746    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:52.511577 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.511548    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:52.541987 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.541962    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:15:52.586951 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.586925    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:15:52.666502 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.666465    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-metrics-certs\") pod \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") "
Sep 16 05:15:52.666732 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.666511    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhjvh\" (UniqueName: \"kubernetes.io/projected/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-kube-api-access-qhjvh\") pod \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") "
Sep 16 05:15:52.666732 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.666707    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-maas-api-tls\") pod \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") "
Sep 16 05:15:52.666880 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.666791    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-openshift-service-ca\") pod \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\" (UID: \"b6b111d3-dfa0-452e-a555-8eb7d8bfa733\") "
Sep 16 05:15:52.667170 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.667149    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b6b111d3-dfa0-452e-a555-8eb7d8bfa733" (UID: "b6b111d3-dfa0-452e-a555-8eb7d8bfa733"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:15:52.669019 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.668995    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b6b111d3-dfa0-452e-a555-8eb7d8bfa733" (UID: "b6b111d3-dfa0-452e-a555-8eb7d8bfa733"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:15:52.669191 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.669167    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b6b111d3-dfa0-452e-a555-8eb7d8bfa733" (UID: "b6b111d3-dfa0-452e-a555-8eb7d8bfa733"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:15:52.669301 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.669192    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-kube-api-access-qhjvh" (OuterVolumeSpecName: "kube-api-access-qhjvh") pod "b6b111d3-dfa0-452e-a555-8eb7d8bfa733" (UID: "b6b111d3-dfa0-452e-a555-8eb7d8bfa733"). InnerVolumeSpecName "kube-api-access-qhjvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:15:52.768312 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.768223    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b820c8-7119-4293-b296-c923d06f2d1b-plugins-config-volume\") pod \"56b820c8-7119-4293-b296-c923d06f2d1b\" (UID: \"56b820c8-7119-4293-b296-c923d06f2d1b\") "
Sep 16 05:15:52.768312 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.768271    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxxtq\" (UniqueName: \"kubernetes.io/projected/56b820c8-7119-4293-b296-c923d06f2d1b-kube-api-access-vxxtq\") pod \"56b820c8-7119-4293-b296-c923d06f2d1b\" (UID: \"56b820c8-7119-4293-b296-c923d06f2d1b\") "
Sep 16 05:15:52.768510 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.768439    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:15:52.768510 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.768451    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhjvh\" (UniqueName: \"kubernetes.io/projected/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-kube-api-access-qhjvh\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:15:52.768510 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.768460    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:15:52.768510 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.768470    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b111d3-dfa0-452e-a555-8eb7d8bfa733-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:15:52.768748 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.768721    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b820c8-7119-4293-b296-c923d06f2d1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56b820c8-7119-4293-b296-c923d06f2d1b" (UID: "56b820c8-7119-4293-b296-c923d06f2d1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:15:52.770530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.770503    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b820c8-7119-4293-b296-c923d06f2d1b-kube-api-access-vxxtq" (OuterVolumeSpecName: "kube-api-access-vxxtq") pod "56b820c8-7119-4293-b296-c923d06f2d1b" (UID: "56b820c8-7119-4293-b296-c923d06f2d1b"). InnerVolumeSpecName "kube-api-access-vxxtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:15:52.835613 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.835569    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:52.869302 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.869272    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b820c8-7119-4293-b296-c923d06f2d1b-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:15:52.869302 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.869298    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxxtq\" (UniqueName: \"kubernetes.io/projected/56b820c8-7119-4293-b296-c923d06f2d1b-kube-api-access-vxxtq\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:15:52.934465 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:52.934435    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:53.221812 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.221777    2573 generic.go:356] "Generic (PLEG): container finished" podID="56b820c8-7119-4293-b296-c923d06f2d1b" containerID="c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312" exitCode=0
Sep 16 05:15:53.222203 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.221839    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"
Sep 16 05:15:53.222203 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.221863    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d" event={"ID":"56b820c8-7119-4293-b296-c923d06f2d1b","Type":"ContainerDied","Data":"c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312"}
Sep 16 05:15:53.222203 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.221945    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d" event={"ID":"56b820c8-7119-4293-b296-c923d06f2d1b","Type":"ContainerDied","Data":"5fddced065fde14314981de9efbb1ee56ac1f9e17cb0d5449cccbabd65dfd2d3"}
Sep 16 05:15:53.222203 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.221976    2573 scope.go:117] "RemoveContainer" containerID="c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312"
Sep 16 05:15:53.223614 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.223567    2573 generic.go:356] "Generic (PLEG): container finished" podID="b6b111d3-dfa0-452e-a555-8eb7d8bfa733" containerID="a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a" exitCode=0
Sep 16 05:15:53.223727 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.223637    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"
Sep 16 05:15:53.223727 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.223651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54" event={"ID":"b6b111d3-dfa0-452e-a555-8eb7d8bfa733","Type":"ContainerDied","Data":"a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a"}
Sep 16 05:15:53.223727 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.223692    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54" event={"ID":"b6b111d3-dfa0-452e-a555-8eb7d8bfa733","Type":"ContainerDied","Data":"1de02f5209b914baa9c0bdd43c9529e8d44ab6abb566a6a2700e8dd1d55d049a"}
Sep 16 05:15:53.231632 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.231614    2573 scope.go:117] "RemoveContainer" containerID="c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312"
Sep 16 05:15:53.231913 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:15:53.231885    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312\": container with ID starting with c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312 not found: ID does not exist" containerID="c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312"
Sep 16 05:15:53.232045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.231922    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312"} err="failed to get container status \"c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312\": rpc error: code = NotFound desc = could not find container \"c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312\": container with ID starting with c2cd36b023c4381c66c42e166293b13b8e45851c034075e75440ebd52699c312 not found: ID does not exist"
Sep 16 05:15:53.232045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.231945    2573 scope.go:117] "RemoveContainer" containerID="a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a"
Sep 16 05:15:53.240566 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.240549    2573 scope.go:117] "RemoveContainer" containerID="a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a"
Sep 16 05:15:53.240901 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:15:53.240881    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a\": container with ID starting with a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a not found: ID does not exist" containerID="a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a"
Sep 16 05:15:53.240975 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.240916    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a"} err="failed to get container status \"a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a\": rpc error: code = NotFound desc = could not find container \"a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a\": container with ID starting with a045053622047227193ca99afb291226ee4675a59376f7db4ffce14edb2f2e8a not found: ID does not exist"
Sep 16 05:15:53.241815 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.241793    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"]
Sep 16 05:15:53.245635 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.245614    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75559a-6dddb74f64-fbv54"]
Sep 16 05:15:53.255850 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.255829    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"]
Sep 16 05:15:53.259327 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.259306    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75559a-6d4bcbd5cd-s297d"]
Sep 16 05:15:53.664326 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:53.664292    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:54.333262 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:54.333226    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:54.786379 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:54.786341    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b820c8-7119-4293-b296-c923d06f2d1b" path="/var/lib/kubelet/pods/56b820c8-7119-4293-b296-c923d06f2d1b/volumes"
Sep 16 05:15:54.786837 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:54.786819    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6b111d3-dfa0-452e-a555-8eb7d8bfa733" path="/var/lib/kubelet/pods/b6b111d3-dfa0-452e-a555-8eb7d8bfa733/volumes"
Sep 16 05:15:55.145340 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:55.145309    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:55.872278 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:55.872248    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:56.645643 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:56.645586    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:57.544148 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:57.544108    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:58.590263 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:58.590229    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:15:58.678856 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:58.678826    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:58.780200 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:58.780167    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:58.929647 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:58.929587    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:15:59.499055 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:15:59.499022    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:00.333507 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:00.333477    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:01.035148 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:01.035118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:01.922214 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:01.922171    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:02.631409 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:02.631377    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:03.311162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:03.311126    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:04.022348 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:04.022310    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:04.643298 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:04.643263    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:06.828702 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:06.828671    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:07.235459 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:07.235426    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:08.453457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:08.453422    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:08.737265 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:08.737186    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:09.416078 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:09.416045    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:10.131067 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:10.131031    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:10.906485 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:10.906450    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:11.555953 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:11.555928    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:11.926111 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:11.926078    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"]
Sep 16 05:16:12.045162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:12.045121    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:14.992924 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:14.992890    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"]
Sep 16 05:16:14.993807 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:14.993372    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8" podUID="7132a6d0-8c3f-4010-941b-085279fd7a97" containerName="maas-api" containerID="cri-o://99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1" gracePeriod=30
Sep 16 05:16:15.019677 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.019651    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:15.056447 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.056417    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"]
Sep 16 05:16:15.056681 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.056656    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb" podUID="6e9f8959-2563-48fb-962b-a882b9aa6f5c" containerName="payload-processing" containerID="cri-o://a85e08592445dc7493b62e4f01821699d3925c48e5021b6ec870dacffb40ca09" gracePeriod=30
Sep 16 05:16:15.061855 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.061827    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"]
Sep 16 05:16:15.062088 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.062047    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj" podUID="238c97f5-b96a-4661-a772-158b0cbf54ad" containerName="payload-pre-processing" containerID="cri-o://4afbc01dde6d4eb5648b09722206d35e6ae120876ccafba9735f00cfa3760298" gracePeriod=30
Sep 16 05:16:15.098768 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.098740    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:15.198471 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.198124    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:15.258290 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.258096    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:15.325031 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.325003    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:16:15.343802 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.343756    2573 generic.go:356] "Generic (PLEG): container finished" podID="238c97f5-b96a-4661-a772-158b0cbf54ad" containerID="4afbc01dde6d4eb5648b09722206d35e6ae120876ccafba9735f00cfa3760298" exitCode=0
Sep 16 05:16:15.343971 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.343809    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj" event={"ID":"238c97f5-b96a-4661-a772-158b0cbf54ad","Type":"ContainerDied","Data":"4afbc01dde6d4eb5648b09722206d35e6ae120876ccafba9735f00cfa3760298"}
Sep 16 05:16:15.346088 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.346062    2573 generic.go:356] "Generic (PLEG): container finished" podID="6e9f8959-2563-48fb-962b-a882b9aa6f5c" containerID="a85e08592445dc7493b62e4f01821699d3925c48e5021b6ec870dacffb40ca09" exitCode=0
Sep 16 05:16:15.346273 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.346217    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb" event={"ID":"6e9f8959-2563-48fb-962b-a882b9aa6f5c","Type":"ContainerDied","Data":"a85e08592445dc7493b62e4f01821699d3925c48e5021b6ec870dacffb40ca09"}
Sep 16 05:16:15.348452 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.348424    2573 generic.go:356] "Generic (PLEG): container finished" podID="7132a6d0-8c3f-4010-941b-085279fd7a97" containerID="99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1" exitCode=0
Sep 16 05:16:15.348584 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.348483    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"
Sep 16 05:16:15.348584 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.348490    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8" event={"ID":"7132a6d0-8c3f-4010-941b-085279fd7a97","Type":"ContainerDied","Data":"99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1"}
Sep 16 05:16:15.348584 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.348538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8" event={"ID":"7132a6d0-8c3f-4010-941b-085279fd7a97","Type":"ContainerDied","Data":"a9197966936b74497e8d8c77fbadab862f65650e961594fcfaa374e9651b59f8"}
Sep 16 05:16:15.348584 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.348570    2573 scope.go:117] "RemoveContainer" containerID="99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1"
Sep 16 05:16:15.356674 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.356650    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:16:15.359383 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.359360    2573 scope.go:117] "RemoveContainer" containerID="99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1"
Sep 16 05:16:15.359783 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:16:15.359694    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1\": container with ID starting with 99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1 not found: ID does not exist" containerID="99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1"
Sep 16 05:16:15.359783 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.359731    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1"} err="failed to get container status \"99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1\": rpc error: code = NotFound desc = could not find container \"99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1\": container with ID starting with 99e3927be12e3ddb84fe31822a915811a2d4787062c02c225b32aa20af724ee1 not found: ID does not exist"
Sep 16 05:16:15.364288 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.364270    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:16:15.376206 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.375419    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-metrics-certs\") pod \"7132a6d0-8c3f-4010-941b-085279fd7a97\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") "
Sep 16 05:16:15.376206 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.375501    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7132a6d0-8c3f-4010-941b-085279fd7a97-openshift-service-ca\") pod \"7132a6d0-8c3f-4010-941b-085279fd7a97\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") "
Sep 16 05:16:15.376206 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.375579    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f75v4\" (UniqueName: \"kubernetes.io/projected/7132a6d0-8c3f-4010-941b-085279fd7a97-kube-api-access-f75v4\") pod \"7132a6d0-8c3f-4010-941b-085279fd7a97\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") "
Sep 16 05:16:15.376206 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.375650    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-maas-api-tls\") pod \"7132a6d0-8c3f-4010-941b-085279fd7a97\" (UID: \"7132a6d0-8c3f-4010-941b-085279fd7a97\") "
Sep 16 05:16:15.377201 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.376992    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7132a6d0-8c3f-4010-941b-085279fd7a97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7132a6d0-8c3f-4010-941b-085279fd7a97" (UID: "7132a6d0-8c3f-4010-941b-085279fd7a97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:16:15.379199 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.379173    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7132a6d0-8c3f-4010-941b-085279fd7a97" (UID: "7132a6d0-8c3f-4010-941b-085279fd7a97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:16:15.383919 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.383894    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7132a6d0-8c3f-4010-941b-085279fd7a97" (UID: "7132a6d0-8c3f-4010-941b-085279fd7a97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:16:15.385194 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.385172    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7132a6d0-8c3f-4010-941b-085279fd7a97-kube-api-access-f75v4" (OuterVolumeSpecName: "kube-api-access-f75v4") pod "7132a6d0-8c3f-4010-941b-085279fd7a97" (UID: "7132a6d0-8c3f-4010-941b-085279fd7a97"). InnerVolumeSpecName "kube-api-access-f75v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:16:15.477281 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.477244    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r9cr\" (UniqueName: \"kubernetes.io/projected/238c97f5-b96a-4661-a772-158b0cbf54ad-kube-api-access-9r9cr\") pod \"238c97f5-b96a-4661-a772-158b0cbf54ad\" (UID: \"238c97f5-b96a-4661-a772-158b0cbf54ad\") "
Sep 16 05:16:15.477419 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.477292    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbftr\" (UniqueName: \"kubernetes.io/projected/6e9f8959-2563-48fb-962b-a882b9aa6f5c-kube-api-access-xbftr\") pod \"6e9f8959-2563-48fb-962b-a882b9aa6f5c\" (UID: \"6e9f8959-2563-48fb-962b-a882b9aa6f5c\") "
Sep 16 05:16:15.477419 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.477319    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9f8959-2563-48fb-962b-a882b9aa6f5c-plugins-config-volume\") pod \"6e9f8959-2563-48fb-962b-a882b9aa6f5c\" (UID: \"6e9f8959-2563-48fb-962b-a882b9aa6f5c\") "
Sep 16 05:16:15.477419 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.477368    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238c97f5-b96a-4661-a772-158b0cbf54ad-plugins-config-volume\") pod \"238c97f5-b96a-4661-a772-158b0cbf54ad\" (UID: \"238c97f5-b96a-4661-a772-158b0cbf54ad\") "
Sep 16 05:16:15.477732 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.477709    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:15.477822 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.477734    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7132a6d0-8c3f-4010-941b-085279fd7a97-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:15.477822 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.477738    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9f8959-2563-48fb-962b-a882b9aa6f5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e9f8959-2563-48fb-962b-a882b9aa6f5c" (UID: "6e9f8959-2563-48fb-962b-a882b9aa6f5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:16:15.477822 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.477749    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7132a6d0-8c3f-4010-941b-085279fd7a97-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:15.477822 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.477781    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/238c97f5-b96a-4661-a772-158b0cbf54ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "238c97f5-b96a-4661-a772-158b0cbf54ad" (UID: "238c97f5-b96a-4661-a772-158b0cbf54ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:16:15.477822 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.477797    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f75v4\" (UniqueName: \"kubernetes.io/projected/7132a6d0-8c3f-4010-941b-085279fd7a97-kube-api-access-f75v4\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:15.479573 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.479540    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9f8959-2563-48fb-962b-a882b9aa6f5c-kube-api-access-xbftr" (OuterVolumeSpecName: "kube-api-access-xbftr") pod "6e9f8959-2563-48fb-962b-a882b9aa6f5c" (UID: "6e9f8959-2563-48fb-962b-a882b9aa6f5c"). InnerVolumeSpecName "kube-api-access-xbftr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:16:15.479573 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.479546    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238c97f5-b96a-4661-a772-158b0cbf54ad-kube-api-access-9r9cr" (OuterVolumeSpecName: "kube-api-access-9r9cr") pod "238c97f5-b96a-4661-a772-158b0cbf54ad" (UID: "238c97f5-b96a-4661-a772-158b0cbf54ad"). InnerVolumeSpecName "kube-api-access-9r9cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:16:15.579000 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.578908    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r9cr\" (UniqueName: \"kubernetes.io/projected/238c97f5-b96a-4661-a772-158b0cbf54ad-kube-api-access-9r9cr\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:15.579000 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.578951    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbftr\" (UniqueName: \"kubernetes.io/projected/6e9f8959-2563-48fb-962b-a882b9aa6f5c-kube-api-access-xbftr\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:15.579000 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.578962    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9f8959-2563-48fb-962b-a882b9aa6f5c-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:15.579000 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.578971    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238c97f5-b96a-4661-a772-158b0cbf54ad-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:15.584370 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.584345    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:15.669549 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.669518    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"]
Sep 16 05:16:15.673715 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:15.673694    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-113278-76c5b5d46f-rw7l8"]
Sep 16 05:16:16.127484 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.127447    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:16.360170 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.360075    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"
Sep 16 05:16:16.360170 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.360091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb" event={"ID":"6e9f8959-2563-48fb-962b-a882b9aa6f5c","Type":"ContainerDied","Data":"a12b219ccd6064ea8d5f57793adfe1b870fe3fb56280559c8dc33ec24b47c92d"}
Sep 16 05:16:16.360411 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.360190    2573 scope.go:117] "RemoveContainer" containerID="a85e08592445dc7493b62e4f01821699d3925c48e5021b6ec870dacffb40ca09"
Sep 16 05:16:16.362581 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.362541    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj" event={"ID":"238c97f5-b96a-4661-a772-158b0cbf54ad","Type":"ContainerDied","Data":"063889493af6d2d81f26bbc826238aac0c4393a49749816a1b3767f08c4c449c"}
Sep 16 05:16:16.362581 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.362560    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"
Sep 16 05:16:16.373030 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.373006    2573 scope.go:117] "RemoveContainer" containerID="4afbc01dde6d4eb5648b09722206d35e6ae120876ccafba9735f00cfa3760298"
Sep 16 05:16:16.388772 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.388743    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"]
Sep 16 05:16:16.392930 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.392898    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-113278-78788d5969-v9rxb"]
Sep 16 05:16:16.404500 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.404478    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"]
Sep 16 05:16:16.408635 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.408615    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-113278-76dc9485c4-dckgj"]
Sep 16 05:16:16.787242 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.787208    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="238c97f5-b96a-4661-a772-158b0cbf54ad" path="/var/lib/kubelet/pods/238c97f5-b96a-4661-a772-158b0cbf54ad/volumes"
Sep 16 05:16:16.787588 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.787576    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e9f8959-2563-48fb-962b-a882b9aa6f5c" path="/var/lib/kubelet/pods/6e9f8959-2563-48fb-962b-a882b9aa6f5c/volumes"
Sep 16 05:16:16.787960 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.787947    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7132a6d0-8c3f-4010-941b-085279fd7a97" path="/var/lib/kubelet/pods/7132a6d0-8c3f-4010-941b-085279fd7a97/volumes"
Sep 16 05:16:16.841878 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:16.841849    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:17.536413 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:17.536378    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:18.350268 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:18.350237    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:19.136057 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:19.136017    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:21.330530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:21.330494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:21.502019 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:21.501984    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:21.625915 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:21.625837    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:21.801310 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:21.801262    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:16:22.357180 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:22.357147    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:22.988770 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:22.988737    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"]
Sep 16 05:16:22.988992 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:22.988969    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5" podUID="a5ea37d0-39e0-42ee-af77-6c233110236d" containerName="maas-api" containerID="cri-o://80694f2b4d4825cd90aac4b5e849749ad2928c340e0c8a8661c87c66a25be5fa" gracePeriod=30
Sep 16 05:16:23.018114 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.018074    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:23.118540 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.118509    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:23.223410 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.223383    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:23.232574 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.232542    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5" podUID="a5ea37d0-39e0-42ee-af77-6c233110236d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.33:8443/health\": dial tcp 10.133.0.33:8443: connect: connection refused"
Sep 16 05:16:23.325378 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.325335    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:23.401835 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.401799    2573 generic.go:356] "Generic (PLEG): container finished" podID="a5ea37d0-39e0-42ee-af77-6c233110236d" containerID="80694f2b4d4825cd90aac4b5e849749ad2928c340e0c8a8661c87c66a25be5fa" exitCode=0
Sep 16 05:16:23.402179 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.401868    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5" event={"ID":"a5ea37d0-39e0-42ee-af77-6c233110236d","Type":"ContainerDied","Data":"80694f2b4d4825cd90aac4b5e849749ad2928c340e0c8a8661c87c66a25be5fa"}
Sep 16 05:16:23.423285 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.423255    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:23.451813 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.451791    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:16:23.551220 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.551135    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ea37d0-39e0-42ee-af77-6c233110236d-openshift-service-ca\") pod \"a5ea37d0-39e0-42ee-af77-6c233110236d\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") "
Sep 16 05:16:23.551220 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.551206    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-maas-api-tls\") pod \"a5ea37d0-39e0-42ee-af77-6c233110236d\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") "
Sep 16 05:16:23.551445 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.551260    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-metrics-certs\") pod \"a5ea37d0-39e0-42ee-af77-6c233110236d\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") "
Sep 16 05:16:23.551445 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.551292    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nns77\" (UniqueName: \"kubernetes.io/projected/a5ea37d0-39e0-42ee-af77-6c233110236d-kube-api-access-nns77\") pod \"a5ea37d0-39e0-42ee-af77-6c233110236d\" (UID: \"a5ea37d0-39e0-42ee-af77-6c233110236d\") "
Sep 16 05:16:23.551592 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.551566    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ea37d0-39e0-42ee-af77-6c233110236d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5ea37d0-39e0-42ee-af77-6c233110236d" (UID: "a5ea37d0-39e0-42ee-af77-6c233110236d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:16:23.553563 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.553535    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5ea37d0-39e0-42ee-af77-6c233110236d" (UID: "a5ea37d0-39e0-42ee-af77-6c233110236d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:16:23.553696 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.553558    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a5ea37d0-39e0-42ee-af77-6c233110236d" (UID: "a5ea37d0-39e0-42ee-af77-6c233110236d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:16:23.553696 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.553563    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ea37d0-39e0-42ee-af77-6c233110236d-kube-api-access-nns77" (OuterVolumeSpecName: "kube-api-access-nns77") pod "a5ea37d0-39e0-42ee-af77-6c233110236d" (UID: "a5ea37d0-39e0-42ee-af77-6c233110236d"). InnerVolumeSpecName "kube-api-access-nns77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:16:23.582467 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.582441    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:23.652095 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.652061    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ea37d0-39e0-42ee-af77-6c233110236d-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:23.652095 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.652095    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:23.652364 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.652110    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5ea37d0-39e0-42ee-af77-6c233110236d-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:23.652364 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:23.652122    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nns77\" (UniqueName: \"kubernetes.io/projected/a5ea37d0-39e0-42ee-af77-6c233110236d-kube-api-access-nns77\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:24.090663 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:24.090631    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:24.409356 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:24.409327    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"
Sep 16 05:16:24.409356 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:24.409342    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5" event={"ID":"a5ea37d0-39e0-42ee-af77-6c233110236d","Type":"ContainerDied","Data":"3c5283cd9f7c14a87fffab489fd9201fe4083c9f5b93e6176086910b1ac784f8"}
Sep 16 05:16:24.409880 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:24.409403    2573 scope.go:117] "RemoveContainer" containerID="80694f2b4d4825cd90aac4b5e849749ad2928c340e0c8a8661c87c66a25be5fa"
Sep 16 05:16:24.430316 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:24.430286    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"]
Sep 16 05:16:24.435031 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:24.435008    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-12b6c6-c49bdf5db-t4ch5"]
Sep 16 05:16:24.787339 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:24.787260    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ea37d0-39e0-42ee-af77-6c233110236d" path="/var/lib/kubelet/pods/a5ea37d0-39e0-42ee-af77-6c233110236d/volumes"
Sep 16 05:16:24.930446 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:24.930405    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:25.771693 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:25.771665    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:26.439985 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:26.439951    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:26.946758 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:26.946723    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n" podUID="4a34a03d-c457-4a5b-aba9-84fb634e469c" containerName="main" containerID="cri-o://d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7" gracePeriod=45
Sep 16 05:16:27.196726 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.196703    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:16:27.280673 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.280586    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-home\") pod \"4a34a03d-c457-4a5b-aba9-84fb634e469c\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") "
Sep 16 05:16:27.280805 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.280716    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a34a03d-c457-4a5b-aba9-84fb634e469c-tls-certs\") pod \"4a34a03d-c457-4a5b-aba9-84fb634e469c\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") "
Sep 16 05:16:27.280805 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.280749    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-model-cache\") pod \"4a34a03d-c457-4a5b-aba9-84fb634e469c\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") "
Sep 16 05:16:27.280805 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.280775    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-dshm\") pod \"4a34a03d-c457-4a5b-aba9-84fb634e469c\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") "
Sep 16 05:16:27.281004 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.280804    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-tmp-dir\") pod \"4a34a03d-c457-4a5b-aba9-84fb634e469c\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") "
Sep 16 05:16:27.281004 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.280824    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-home" (OuterVolumeSpecName: "home") pod "4a34a03d-c457-4a5b-aba9-84fb634e469c" (UID: "4a34a03d-c457-4a5b-aba9-84fb634e469c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:27.281004 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.280865    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svldr\" (UniqueName: \"kubernetes.io/projected/4a34a03d-c457-4a5b-aba9-84fb634e469c-kube-api-access-svldr\") pod \"4a34a03d-c457-4a5b-aba9-84fb634e469c\" (UID: \"4a34a03d-c457-4a5b-aba9-84fb634e469c\") "
Sep 16 05:16:27.281169 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.280995    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a34a03d-c457-4a5b-aba9-84fb634e469c" (UID: "4a34a03d-c457-4a5b-aba9-84fb634e469c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:27.281169 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.281110    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a34a03d-c457-4a5b-aba9-84fb634e469c" (UID: "4a34a03d-c457-4a5b-aba9-84fb634e469c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:27.281169 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.281165    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:27.281335 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.281180    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:27.282985 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.282962    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-dshm" (OuterVolumeSpecName: "dshm") pod "4a34a03d-c457-4a5b-aba9-84fb634e469c" (UID: "4a34a03d-c457-4a5b-aba9-84fb634e469c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:27.282985 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.282969    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a34a03d-c457-4a5b-aba9-84fb634e469c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a34a03d-c457-4a5b-aba9-84fb634e469c" (UID: "4a34a03d-c457-4a5b-aba9-84fb634e469c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:16:27.283140 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.283031    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a34a03d-c457-4a5b-aba9-84fb634e469c-kube-api-access-svldr" (OuterVolumeSpecName: "kube-api-access-svldr") pod "4a34a03d-c457-4a5b-aba9-84fb634e469c" (UID: "4a34a03d-c457-4a5b-aba9-84fb634e469c"). InnerVolumeSpecName "kube-api-access-svldr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:16:27.380568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.380532    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:27.381717 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.381697    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:27.381783 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.381719    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svldr\" (UniqueName: \"kubernetes.io/projected/4a34a03d-c457-4a5b-aba9-84fb634e469c-kube-api-access-svldr\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:27.381783 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.381728    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a34a03d-c457-4a5b-aba9-84fb634e469c-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:27.381783 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.381736    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a34a03d-c457-4a5b-aba9-84fb634e469c-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:27.425864 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.425829    2573 generic.go:356] "Generic (PLEG): container finished" podID="4a34a03d-c457-4a5b-aba9-84fb634e469c" containerID="d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7" exitCode=0
Sep 16 05:16:27.426039 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.425885    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"
Sep 16 05:16:27.426039 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.425894    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n" event={"ID":"4a34a03d-c457-4a5b-aba9-84fb634e469c","Type":"ContainerDied","Data":"d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7"}
Sep 16 05:16:27.426039 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.425929    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n" event={"ID":"4a34a03d-c457-4a5b-aba9-84fb634e469c","Type":"ContainerDied","Data":"5488387355fd28dbf11521bd08bce569084add5e6deaf32036401bd8d519366d"}
Sep 16 05:16:27.426039 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.425946    2573 scope.go:117] "RemoveContainer" containerID="d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7"
Sep 16 05:16:27.434701 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.434682    2573 scope.go:117] "RemoveContainer" containerID="d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7"
Sep 16 05:16:27.435003 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:16:27.434977    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7\": container with ID starting with d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7 not found: ID does not exist" containerID="d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7"
Sep 16 05:16:27.435123 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.435012    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7"} err="failed to get container status \"d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7\": rpc error: code = NotFound desc = could not find container \"d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7\": container with ID starting with d8dcaa39d632927587606cf07390053f2026a7f35a27626e9879b04e6b30f3f7 not found: ID does not exist"
Sep 16 05:16:27.458510 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.458435    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"]
Sep 16 05:16:27.463547 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:27.463516    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/distinct-w4-12b6c6-kserve-6c7d864cc-gcv8n"]
Sep 16 05:16:28.786720 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:28.786684    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a34a03d-c457-4a5b-aba9-84fb634e469c" path="/var/lib/kubelet/pods/4a34a03d-c457-4a5b-aba9-84fb634e469c/volumes"
Sep 16 05:16:29.272541 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:29.272510    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:29.373119 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:29.373084    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:29.735835 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:29.735804    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:32.162652 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:32.162582    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:33.063188 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.063160    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:33.104395 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.104360    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"]
Sep 16 05:16:33.104643 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.104617    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v" podUID="83c190d7-6080-402d-b848-e4ee934374ee" containerName="payload-processing" containerID="cri-o://6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e" gracePeriod=30
Sep 16 05:16:33.159548 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.159520    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:33.259260 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.259230    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:33.358804 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.358773    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:16:33.375848 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.375810    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:33.456720 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.456689    2573 generic.go:356] "Generic (PLEG): container finished" podID="83c190d7-6080-402d-b848-e4ee934374ee" containerID="6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e" exitCode=0
Sep 16 05:16:33.456900 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.456736    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"
Sep 16 05:16:33.456900 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.456775    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v" event={"ID":"83c190d7-6080-402d-b848-e4ee934374ee","Type":"ContainerDied","Data":"6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e"}
Sep 16 05:16:33.456900 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.456830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v" event={"ID":"83c190d7-6080-402d-b848-e4ee934374ee","Type":"ContainerDied","Data":"2ad39f9fe7dbe154c730d3c4c459abcffc04e4a4b4d74a546d6601aca9a7115b"}
Sep 16 05:16:33.456900 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.456860    2573 scope.go:117] "RemoveContainer" containerID="6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e"
Sep 16 05:16:33.465313 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.465296    2573 scope.go:117] "RemoveContainer" containerID="6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e"
Sep 16 05:16:33.465631 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:16:33.465613    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e\": container with ID starting with 6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e not found: ID does not exist" containerID="6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e"
Sep 16 05:16:33.465692 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.465638    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e"} err="failed to get container status \"6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e\": rpc error: code = NotFound desc = could not find container \"6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e\": container with ID starting with 6ac69485425c58bfbf4913b5099fc1f353a903fccffb7a6dca63115450c6b13e not found: ID does not exist"
Sep 16 05:16:33.468458 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.468433    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:33.540914 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.540884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c190d7-6080-402d-b848-e4ee934374ee-plugins-config-volume\") pod \"83c190d7-6080-402d-b848-e4ee934374ee\" (UID: \"83c190d7-6080-402d-b848-e4ee934374ee\") "
Sep 16 05:16:33.541077 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.540974    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjclz\" (UniqueName: \"kubernetes.io/projected/83c190d7-6080-402d-b848-e4ee934374ee-kube-api-access-jjclz\") pod \"83c190d7-6080-402d-b848-e4ee934374ee\" (UID: \"83c190d7-6080-402d-b848-e4ee934374ee\") "
Sep 16 05:16:33.541256 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.541232    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c190d7-6080-402d-b848-e4ee934374ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "83c190d7-6080-402d-b848-e4ee934374ee" (UID: "83c190d7-6080-402d-b848-e4ee934374ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:16:33.543042 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.543019    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c190d7-6080-402d-b848-e4ee934374ee-kube-api-access-jjclz" (OuterVolumeSpecName: "kube-api-access-jjclz") pod "83c190d7-6080-402d-b848-e4ee934374ee" (UID: "83c190d7-6080-402d-b848-e4ee934374ee"). InnerVolumeSpecName "kube-api-access-jjclz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:16:33.641933 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.641904    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjclz\" (UniqueName: \"kubernetes.io/projected/83c190d7-6080-402d-b848-e4ee934374ee-kube-api-access-jjclz\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:33.641933 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.641934    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c190d7-6080-402d-b848-e4ee934374ee-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:33.745977 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.745941    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:33.784875 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.784846    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"]
Sep 16 05:16:33.789934 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.789910    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad06de-9d54d8b45-7h44v"]
Sep 16 05:16:33.844459 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:33.844418    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:34.453264 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:34.453222    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:34.786589 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:34.786509    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c190d7-6080-402d-b848-e4ee934374ee" path="/var/lib/kubelet/pods/83c190d7-6080-402d-b848-e4ee934374ee/volumes"
Sep 16 05:16:35.198076 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:35.198038    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:36.231947 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:36.231917    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:37.778613 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:37.778563    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:16:37.900706 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:37.900677    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"]
Sep 16 05:16:37.918624 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:37.918580    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:38.119342 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:38.119258    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:38.174740 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:38.174712    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:39.548920 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:39.548888    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:39.615077 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:39.615042    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:39.803124 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:39.803038    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:40.237322 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:40.237285    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"]
Sep 16 05:16:40.258408 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:40.258375    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:40.410377 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:40.410344    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:41.084423 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:41.084392    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:42.020898 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:42.020868    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:42.750382 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:42.750349    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:43.566765 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:43.566730    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:45.116776 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:45.116748    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:45.175893 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:45.175861    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:48.154932 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:48.154894    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:48.221556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:48.221526    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:49.592858 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:49.592822    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:50.680918 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.680881    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"]
Sep 16 05:16:50.681345 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.681126    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x" podUID="2047c935-7175-4fae-b9ad-fd8c92b71c09" containerName="maas-api" containerID="cri-o://d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba" gracePeriod=30
Sep 16 05:16:50.714490 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.714459    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:50.792801 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.792767    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:50.868393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.868346    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:50.931018 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.930955    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:16:50.938202 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.938177    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:50.985530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.985504    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpblz\" (UniqueName: \"kubernetes.io/projected/2047c935-7175-4fae-b9ad-fd8c92b71c09-kube-api-access-lpblz\") pod \"2047c935-7175-4fae-b9ad-fd8c92b71c09\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") "
Sep 16 05:16:50.985720 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.985550    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-maas-api-tls\") pod \"2047c935-7175-4fae-b9ad-fd8c92b71c09\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") "
Sep 16 05:16:50.985720 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.985575    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2047c935-7175-4fae-b9ad-fd8c92b71c09-openshift-service-ca\") pod \"2047c935-7175-4fae-b9ad-fd8c92b71c09\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") "
Sep 16 05:16:50.985720 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.985623    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-metrics-certs\") pod \"2047c935-7175-4fae-b9ad-fd8c92b71c09\" (UID: \"2047c935-7175-4fae-b9ad-fd8c92b71c09\") "
Sep 16 05:16:50.985985 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.985959    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2047c935-7175-4fae-b9ad-fd8c92b71c09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2047c935-7175-4fae-b9ad-fd8c92b71c09" (UID: "2047c935-7175-4fae-b9ad-fd8c92b71c09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:16:50.987818 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.987796    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2047c935-7175-4fae-b9ad-fd8c92b71c09" (UID: "2047c935-7175-4fae-b9ad-fd8c92b71c09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:16:50.987897 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.987836    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2047c935-7175-4fae-b9ad-fd8c92b71c09" (UID: "2047c935-7175-4fae-b9ad-fd8c92b71c09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:16:50.987897 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:50.987873    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2047c935-7175-4fae-b9ad-fd8c92b71c09-kube-api-access-lpblz" (OuterVolumeSpecName: "kube-api-access-lpblz") pod "2047c935-7175-4fae-b9ad-fd8c92b71c09" (UID: "2047c935-7175-4fae-b9ad-fd8c92b71c09"). InnerVolumeSpecName "kube-api-access-lpblz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:16:51.086492 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.086459    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpblz\" (UniqueName: \"kubernetes.io/projected/2047c935-7175-4fae-b9ad-fd8c92b71c09-kube-api-access-lpblz\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:51.086492 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.086490    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:51.086492 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.086499    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2047c935-7175-4fae-b9ad-fd8c92b71c09-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:51.086742 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.086507    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2047c935-7175-4fae-b9ad-fd8c92b71c09-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:51.269122 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.269037    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:51.366784 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.366750    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:51.384679 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.384656    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:16:51.544285 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.544210    2573 generic.go:356] "Generic (PLEG): container finished" podID="2047c935-7175-4fae-b9ad-fd8c92b71c09" containerID="d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba" exitCode=0
Sep 16 05:16:51.544285 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.544253    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"
Sep 16 05:16:51.544474 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.544305    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x" event={"ID":"2047c935-7175-4fae-b9ad-fd8c92b71c09","Type":"ContainerDied","Data":"d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba"}
Sep 16 05:16:51.544474 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.544361    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x" event={"ID":"2047c935-7175-4fae-b9ad-fd8c92b71c09","Type":"ContainerDied","Data":"2ab598594e430fce377b24fef0d0d8d4cb032c99217b310a5c425c612f99cade"}
Sep 16 05:16:51.544474 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.544384    2573 scope.go:117] "RemoveContainer" containerID="d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba"
Sep 16 05:16:51.554171 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.554151    2573 scope.go:117] "RemoveContainer" containerID="d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba"
Sep 16 05:16:51.554455 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:16:51.554435    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba\": container with ID starting with d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba not found: ID does not exist" containerID="d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba"
Sep 16 05:16:51.554501 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.554464    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba"} err="failed to get container status \"d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba\": rpc error: code = NotFound desc = could not find container \"d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba\": container with ID starting with d114db944464bc6f411f1bd27266b53405be4a0e35eba7305d90ec2c04e1d2ba not found: ID does not exist"
Sep 16 05:16:51.572487 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.572464    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"]
Sep 16 05:16:51.593434 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:51.593410    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ee234-85f789df77-d996x"]
Sep 16 05:16:52.250585 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:52.250551    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:52.785490 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:52.785457    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2047c935-7175-4fae-b9ad-fd8c92b71c09" path="/var/lib/kubelet/pods/2047c935-7175-4fae-b9ad-fd8c92b71c09/volumes"
Sep 16 05:16:52.919921 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:52.919874    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9" podUID="12c97f6e-5742-4f0d-b6aa-826e35634677" containerName="main" containerID="cri-o://489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0" gracePeriod=45
Sep 16 05:16:52.951727 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:52.951694    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:53.165915 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.165890    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:16:53.202876 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.202848    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-home\") pod \"12c97f6e-5742-4f0d-b6aa-826e35634677\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") "
Sep 16 05:16:53.203024 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.202886    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c97f6e-5742-4f0d-b6aa-826e35634677-tls-certs\") pod \"12c97f6e-5742-4f0d-b6aa-826e35634677\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") "
Sep 16 05:16:53.203024 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.202993    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-tmp-dir\") pod \"12c97f6e-5742-4f0d-b6aa-826e35634677\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") "
Sep 16 05:16:53.203114 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.203022    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zpkb\" (UniqueName: \"kubernetes.io/projected/12c97f6e-5742-4f0d-b6aa-826e35634677-kube-api-access-5zpkb\") pod \"12c97f6e-5742-4f0d-b6aa-826e35634677\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") "
Sep 16 05:16:53.203114 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.203047    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-dshm\") pod \"12c97f6e-5742-4f0d-b6aa-826e35634677\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") "
Sep 16 05:16:53.203114 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.203072    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-model-cache\") pod \"12c97f6e-5742-4f0d-b6aa-826e35634677\" (UID: \"12c97f6e-5742-4f0d-b6aa-826e35634677\") "
Sep 16 05:16:53.203263 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.203137    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-home" (OuterVolumeSpecName: "home") pod "12c97f6e-5742-4f0d-b6aa-826e35634677" (UID: "12c97f6e-5742-4f0d-b6aa-826e35634677"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:53.203322 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.203266    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12c97f6e-5742-4f0d-b6aa-826e35634677" (UID: "12c97f6e-5742-4f0d-b6aa-826e35634677"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:53.203384 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.203343    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-model-cache" (OuterVolumeSpecName: "model-cache") pod "12c97f6e-5742-4f0d-b6aa-826e35634677" (UID: "12c97f6e-5742-4f0d-b6aa-826e35634677"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:53.203458 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.203442    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:53.203499 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.203464    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:53.203499 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.203476    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:53.205260 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.205237    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c97f6e-5742-4f0d-b6aa-826e35634677-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12c97f6e-5742-4f0d-b6aa-826e35634677" (UID: "12c97f6e-5742-4f0d-b6aa-826e35634677"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:16:53.205379 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.205308    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c97f6e-5742-4f0d-b6aa-826e35634677-kube-api-access-5zpkb" (OuterVolumeSpecName: "kube-api-access-5zpkb") pod "12c97f6e-5742-4f0d-b6aa-826e35634677" (UID: "12c97f6e-5742-4f0d-b6aa-826e35634677"). InnerVolumeSpecName "kube-api-access-5zpkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:16:53.205379 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.205322    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-dshm" (OuterVolumeSpecName: "dshm") pod "12c97f6e-5742-4f0d-b6aa-826e35634677" (UID: "12c97f6e-5742-4f0d-b6aa-826e35634677"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:53.303881 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.303847    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c97f6e-5742-4f0d-b6aa-826e35634677-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:53.303881 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.303874    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c97f6e-5742-4f0d-b6aa-826e35634677-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:53.303881 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.303888    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zpkb\" (UniqueName: \"kubernetes.io/projected/12c97f6e-5742-4f0d-b6aa-826e35634677-kube-api-access-5zpkb\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:53.556942 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.556908    2573 generic.go:356] "Generic (PLEG): container finished" podID="12c97f6e-5742-4f0d-b6aa-826e35634677" containerID="489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0" exitCode=0
Sep 16 05:16:53.557147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.556957    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"
Sep 16 05:16:53.557147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.556992    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9" event={"ID":"12c97f6e-5742-4f0d-b6aa-826e35634677","Type":"ContainerDied","Data":"489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0"}
Sep 16 05:16:53.557147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.557038    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9" event={"ID":"12c97f6e-5742-4f0d-b6aa-826e35634677","Type":"ContainerDied","Data":"64afe26c52e32d0448e397d161777ecc54ae4f6340cea6fa5eeaeb8b06941b5c"}
Sep 16 05:16:53.557147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.557056    2573 scope.go:117] "RemoveContainer" containerID="489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0"
Sep 16 05:16:53.566103 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.566084    2573 scope.go:117] "RemoveContainer" containerID="489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0"
Sep 16 05:16:53.566362 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:16:53.566342    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0\": container with ID starting with 489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0 not found: ID does not exist" containerID="489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0"
Sep 16 05:16:53.566427 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.566374    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0"} err="failed to get container status \"489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0\": rpc error: code = NotFound desc = could not find container \"489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0\": container with ID starting with 489ef2bb46f69a140de08284c307e64c7e4690da524ead16a56fd677e06d9fe0 not found: ID does not exist"
Sep 16 05:16:53.577035 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.577012    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"]
Sep 16 05:16:53.581868 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:53.581848    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/auth-test-model-3784e2-kserve-64d55f59b7-wq5l9"]
Sep 16 05:16:54.786029 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:54.785994    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12c97f6e-5742-4f0d-b6aa-826e35634677" path="/var/lib/kubelet/pods/12c97f6e-5742-4f0d-b6aa-826e35634677/volumes"
Sep 16 05:16:55.260621 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.260557    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg" podUID="7e59d152-ce8d-4e87-9f4e-31370462f72d" containerName="main" containerID="cri-o://8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d" gracePeriod=45
Sep 16 05:16:55.510728 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.510666    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:16:55.569533 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.569496    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e59d152-ce8d-4e87-9f4e-31370462f72d" containerID="8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d" exitCode=0
Sep 16 05:16:55.569706 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.569537    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"
Sep 16 05:16:55.569706 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.569549    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg" event={"ID":"7e59d152-ce8d-4e87-9f4e-31370462f72d","Type":"ContainerDied","Data":"8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d"}
Sep 16 05:16:55.569706 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.569581    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg" event={"ID":"7e59d152-ce8d-4e87-9f4e-31370462f72d","Type":"ContainerDied","Data":"5e158a7e001bb4b32d293bccc7f04ee618c3d104e155442d06efe423f2720e85"}
Sep 16 05:16:55.569706 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.569616    2573 scope.go:117] "RemoveContainer" containerID="8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d"
Sep 16 05:16:55.579549 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.579515    2573 scope.go:117] "RemoveContainer" containerID="8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d"
Sep 16 05:16:55.579933 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:16:55.579914    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d\": container with ID starting with 8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d not found: ID does not exist" containerID="8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d"
Sep 16 05:16:55.580014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.579943    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d"} err="failed to get container status \"8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d\": rpc error: code = NotFound desc = could not find container \"8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d\": container with ID starting with 8a2a0195cd421df06403a8627ccfda09e25b138526caae5ded3a41d89fb7151d not found: ID does not exist"
Sep 16 05:16:55.626449 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.626413    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t565f\" (UniqueName: \"kubernetes.io/projected/7e59d152-ce8d-4e87-9f4e-31370462f72d-kube-api-access-t565f\") pod \"7e59d152-ce8d-4e87-9f4e-31370462f72d\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") "
Sep 16 05:16:55.626627 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.626466    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e59d152-ce8d-4e87-9f4e-31370462f72d-tls-certs\") pod \"7e59d152-ce8d-4e87-9f4e-31370462f72d\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") "
Sep 16 05:16:55.626627 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.626525    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-tmp-dir\") pod \"7e59d152-ce8d-4e87-9f4e-31370462f72d\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") "
Sep 16 05:16:55.626627 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.626559    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-home\") pod \"7e59d152-ce8d-4e87-9f4e-31370462f72d\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") "
Sep 16 05:16:55.626627 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.626580    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-dshm\") pod \"7e59d152-ce8d-4e87-9f4e-31370462f72d\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") "
Sep 16 05:16:55.626852 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.626633    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-model-cache\") pod \"7e59d152-ce8d-4e87-9f4e-31370462f72d\" (UID: \"7e59d152-ce8d-4e87-9f4e-31370462f72d\") "
Sep 16 05:16:55.626906 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.626863    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e59d152-ce8d-4e87-9f4e-31370462f72d" (UID: "7e59d152-ce8d-4e87-9f4e-31370462f72d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:55.627119 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.627093    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e59d152-ce8d-4e87-9f4e-31370462f72d" (UID: "7e59d152-ce8d-4e87-9f4e-31370462f72d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:55.627183 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.627112    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-home" (OuterVolumeSpecName: "home") pod "7e59d152-ce8d-4e87-9f4e-31370462f72d" (UID: "7e59d152-ce8d-4e87-9f4e-31370462f72d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:55.628880 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.628849    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e59d152-ce8d-4e87-9f4e-31370462f72d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e59d152-ce8d-4e87-9f4e-31370462f72d" (UID: "7e59d152-ce8d-4e87-9f4e-31370462f72d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:16:55.629120 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.629098    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-dshm" (OuterVolumeSpecName: "dshm") pod "7e59d152-ce8d-4e87-9f4e-31370462f72d" (UID: "7e59d152-ce8d-4e87-9f4e-31370462f72d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:16:55.629181 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.629103    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e59d152-ce8d-4e87-9f4e-31370462f72d-kube-api-access-t565f" (OuterVolumeSpecName: "kube-api-access-t565f") pod "7e59d152-ce8d-4e87-9f4e-31370462f72d" (UID: "7e59d152-ce8d-4e87-9f4e-31370462f72d"). InnerVolumeSpecName "kube-api-access-t565f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:16:55.727530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.727498    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e59d152-ce8d-4e87-9f4e-31370462f72d-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:55.727530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.727523    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:55.727530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.727536    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:55.727753 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.727546    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:55.727753 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.727557    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e59d152-ce8d-4e87-9f4e-31370462f72d-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:55.727753 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.727569    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t565f\" (UniqueName: \"kubernetes.io/projected/7e59d152-ce8d-4e87-9f4e-31370462f72d-kube-api-access-t565f\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:16:55.890890 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.890856    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"]
Sep 16 05:16:55.892997 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:55.892973    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/auth-test-model-10ee79-kserve-7d96c94599-l2wlg"]
Sep 16 05:16:56.786131 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:56.786096    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e59d152-ce8d-4e87-9f4e-31370462f72d" path="/var/lib/kubelet/pods/7e59d152-ce8d-4e87-9f4e-31370462f72d/volumes"
Sep 16 05:16:57.460748 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:57.460720    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:57.561043 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:57.561007    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:16:57.711012 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:16:57.710936    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:17:00.875252 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:00.875213    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:17:15.388546 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:15.388513    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:17:18.883404 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:18.883366    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:17:21.807765 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:21.807683    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:17:28.695509 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:28.695477    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:17:44.553426 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553386    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"]
Sep 16 05:17:44.553846 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553823    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7132a6d0-8c3f-4010-941b-085279fd7a97" containerName="maas-api"
Sep 16 05:17:44.553846 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553836    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7132a6d0-8c3f-4010-941b-085279fd7a97" containerName="maas-api"
Sep 16 05:17:44.553846 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553845    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2047c935-7175-4fae-b9ad-fd8c92b71c09" containerName="maas-api"
Sep 16 05:17:44.553846 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553850    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2047c935-7175-4fae-b9ad-fd8c92b71c09" containerName="maas-api"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553861    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ea37d0-39e0-42ee-af77-6c233110236d" containerName="maas-api"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553866    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ea37d0-39e0-42ee-af77-6c233110236d" containerName="maas-api"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553875    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a34a03d-c457-4a5b-aba9-84fb634e469c" containerName="main"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553880    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a34a03d-c457-4a5b-aba9-84fb634e469c" containerName="main"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553888    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b820c8-7119-4293-b296-c923d06f2d1b" containerName="payload-pre-processing"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553895    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b820c8-7119-4293-b296-c923d06f2d1b" containerName="payload-pre-processing"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553903    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c190d7-6080-402d-b848-e4ee934374ee" containerName="payload-processing"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553908    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c190d7-6080-402d-b848-e4ee934374ee" containerName="payload-processing"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553915    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e59d152-ce8d-4e87-9f4e-31370462f72d" containerName="main"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553920    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e59d152-ce8d-4e87-9f4e-31370462f72d" containerName="main"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553927    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e9f8959-2563-48fb-962b-a882b9aa6f5c" containerName="payload-processing"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553932    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e9f8959-2563-48fb-962b-a882b9aa6f5c" containerName="payload-processing"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553941    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="238c97f5-b96a-4661-a772-158b0cbf54ad" containerName="payload-pre-processing"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553946    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="238c97f5-b96a-4661-a772-158b0cbf54ad" containerName="payload-pre-processing"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553954    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c97f6e-5742-4f0d-b6aa-826e35634677" containerName="main"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553958    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c97f6e-5742-4f0d-b6aa-826e35634677" containerName="main"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553964    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6b111d3-dfa0-452e-a555-8eb7d8bfa733" containerName="maas-api"
Sep 16 05:17:44.553996 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.553969    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6b111d3-dfa0-452e-a555-8eb7d8bfa733" containerName="maas-api"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554016    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e9f8959-2563-48fb-962b-a882b9aa6f5c" containerName="payload-processing"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554024    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ea37d0-39e0-42ee-af77-6c233110236d" containerName="maas-api"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554029    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b820c8-7119-4293-b296-c923d06f2d1b" containerName="payload-pre-processing"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554035    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a34a03d-c457-4a5b-aba9-84fb634e469c" containerName="main"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554041    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c97f6e-5742-4f0d-b6aa-826e35634677" containerName="main"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554046    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e59d152-ce8d-4e87-9f4e-31370462f72d" containerName="main"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554051    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2047c935-7175-4fae-b9ad-fd8c92b71c09" containerName="maas-api"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554056    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="238c97f5-b96a-4661-a772-158b0cbf54ad" containerName="payload-pre-processing"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554062    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c190d7-6080-402d-b848-e4ee934374ee" containerName="payload-processing"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554067    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7132a6d0-8c3f-4010-941b-085279fd7a97" containerName="maas-api"
Sep 16 05:17:44.554790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.554072    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6b111d3-dfa0-452e-a555-8eb7d8bfa733" containerName="maas-api"
Sep 16 05:17:44.557071 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.557049    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.559199 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.559176    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-10ee79\"/\"openshift-service-ca.crt\""
Sep 16 05:17:44.559983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.559958    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-10ee79\"/\"sub-test-model-10ee79-kserve-self-signed-certs\""
Sep 16 05:17:44.559983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.559978    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-10ee79\"/\"default-dockercfg-7dskn\""
Sep 16 05:17:44.560148 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.559963    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-10ee79\"/\"kube-root-ca.crt\""
Sep 16 05:17:44.567372 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.567343    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"]
Sep 16 05:17:44.589219 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.589189    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:17:44.637762 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.637715    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-model-cache\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.637762 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.637771    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-dshm\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.638011 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.637859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bs6\" (UniqueName: \"kubernetes.io/projected/450dde36-b462-4b8d-a695-ff90e1fc4da9-kube-api-access-67bs6\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.638011 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.637911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-home\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.638011 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.637963    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450dde36-b462-4b8d-a695-ff90e1fc4da9-tls-certs\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.638118 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.638023    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-tmp-dir\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.738410 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.738363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bs6\" (UniqueName: \"kubernetes.io/projected/450dde36-b462-4b8d-a695-ff90e1fc4da9-kube-api-access-67bs6\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.738410 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.738416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-home\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.738719 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.738438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450dde36-b462-4b8d-a695-ff90e1fc4da9-tls-certs\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.738719 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.738457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-tmp-dir\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.738719 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.738495    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-model-cache\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.738719 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.738528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-dshm\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.738937 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.738895    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-home\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.738997 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.738932    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-tmp-dir\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.738997 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.738959    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-model-cache\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.740947 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.740920    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-dshm\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.741196 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.741176    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450dde36-b462-4b8d-a695-ff90e1fc4da9-tls-certs\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.746297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.746269    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bs6\" (UniqueName: \"kubernetes.io/projected/450dde36-b462-4b8d-a695-ff90e1fc4da9-kube-api-access-67bs6\") pod \"sub-test-model-10ee79-kserve-6686b98fbf-dtd5k\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") " pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:44.867858 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:44.867761    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:45.144894 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:45.144870    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"]
Sep 16 05:17:45.147659 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:17:45.147626    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod450dde36_b462_4b8d_a695_ff90e1fc4da9.slice/crio-a45bcbcb21ebd1929e5e145aebc187bd166f0484f5bef2ef861d598c7675638f WatchSource:0}: Error finding container a45bcbcb21ebd1929e5e145aebc187bd166f0484f5bef2ef861d598c7675638f: Status 404 returned error can't find the container with id a45bcbcb21ebd1929e5e145aebc187bd166f0484f5bef2ef861d598c7675638f
Sep 16 05:17:45.803493 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:45.803457    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k" event={"ID":"450dde36-b462-4b8d-a695-ff90e1fc4da9","Type":"ContainerStarted","Data":"a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776"}
Sep 16 05:17:45.804042 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:45.803504    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k" event={"ID":"450dde36-b462-4b8d-a695-ff90e1fc4da9","Type":"ContainerStarted","Data":"a45bcbcb21ebd1929e5e145aebc187bd166f0484f5bef2ef861d598c7675638f"}
Sep 16 05:17:45.821138 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:45.821085    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k" podStartSLOduration=1.821067046 podStartE2EDuration="1.821067046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:17:45.820064976 +0000 UTC m=+1729.485994568" watchObservedRunningTime="2026-09-16 05:17:45.821067046 +0000 UTC m=+1729.486996643"
Sep 16 05:17:48.278101 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:48.278054    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:17:54.868435 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:54.868395    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:54.868435 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:54.868444    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:54.881506 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:54.881477    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:17:55.865798 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:17:55.865766    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:18:13.881766 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:13.881729    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:18:26.578777 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:26.578734    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:18:36.487522 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:36.487487    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:18:37.255328 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:37.255291    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"]
Sep 16 05:18:37.274698 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:37.274665    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:18:40.380187 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:40.380150    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:18:49.533810 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.533777    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"]
Sep 16 05:18:49.537253 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.537229    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.540029 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.540003    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3784e2\"/\"openshift-service-ca.crt\""
Sep 16 05:18:49.540163 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.540048    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3784e2\"/\"rate-test-model-3784e2-kserve-self-signed-certs\""
Sep 16 05:18:49.540163 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.540048    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3784e2\"/\"kube-root-ca.crt\""
Sep 16 05:18:49.540163 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.540118    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3784e2\"/\"default-dockercfg-zr4fr\""
Sep 16 05:18:49.548086 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.548062    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"]
Sep 16 05:18:49.575759 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.575727    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-home\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.575759 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.575759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-tmp-dir\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.575992 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.575785    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-dshm\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.575992 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.575820    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fdc426-e1fe-48fe-8307-df8ecfc79203-tls-certs\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.575992 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.575839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-model-cache\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.575992 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.575912    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4w5\" (UniqueName: \"kubernetes.io/projected/78fdc426-e1fe-48fe-8307-df8ecfc79203-kube-api-access-dn4w5\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.581323 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.581299    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:18:49.676347 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.676267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-home\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.676347 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.676307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-tmp-dir\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.676556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.676348    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-dshm\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.676556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.676379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fdc426-e1fe-48fe-8307-df8ecfc79203-tls-certs\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.676556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.676406    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-model-cache\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.676556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.676466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4w5\" (UniqueName: \"kubernetes.io/projected/78fdc426-e1fe-48fe-8307-df8ecfc79203-kube-api-access-dn4w5\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.676801 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.676741    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-home\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.676801 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.676780    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-tmp-dir\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.676888 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.676858    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-model-cache\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.678746 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.678716    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-dshm\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.678989 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.678973    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fdc426-e1fe-48fe-8307-df8ecfc79203-tls-certs\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.685210 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.685184    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4w5\" (UniqueName: \"kubernetes.io/projected/78fdc426-e1fe-48fe-8307-df8ecfc79203-kube-api-access-dn4w5\") pod \"rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") " pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:49.847294 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:49.847208    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:50.120113 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:50.120019    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"]
Sep 16 05:18:50.124640 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:18:50.124578    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78fdc426_e1fe_48fe_8307_df8ecfc79203.slice/crio-bcba237ef571659c81514e158aee38047a028c797dc40685a06b5877cccc61b5 WatchSource:0}: Error finding container bcba237ef571659c81514e158aee38047a028c797dc40685a06b5877cccc61b5: Status 404 returned error can't find the container with id bcba237ef571659c81514e158aee38047a028c797dc40685a06b5877cccc61b5
Sep 16 05:18:51.107710 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:51.107676    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6" event={"ID":"78fdc426-e1fe-48fe-8307-df8ecfc79203","Type":"ContainerStarted","Data":"e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b"}
Sep 16 05:18:51.107710 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:51.107720    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6" event={"ID":"78fdc426-e1fe-48fe-8307-df8ecfc79203","Type":"ContainerStarted","Data":"bcba237ef571659c81514e158aee38047a028c797dc40685a06b5877cccc61b5"}
Sep 16 05:18:51.124339 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:51.124282    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6" podStartSLOduration=2.124267925 podStartE2EDuration="2.124267925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:18:51.122505414 +0000 UTC m=+1794.788435007" watchObservedRunningTime="2026-09-16 05:18:51.124267925 +0000 UTC m=+1794.790197518"
Sep 16 05:18:52.274192 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.274150    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k" podUID="450dde36-b462-4b8d-a695-ff90e1fc4da9" containerName="main" containerID="cri-o://a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776" gracePeriod=45
Sep 16 05:18:52.519735 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.519708    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:18:52.597567 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.597489    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450dde36-b462-4b8d-a695-ff90e1fc4da9-tls-certs\") pod \"450dde36-b462-4b8d-a695-ff90e1fc4da9\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") "
Sep 16 05:18:52.597567 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.597539    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-dshm\") pod \"450dde36-b462-4b8d-a695-ff90e1fc4da9\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") "
Sep 16 05:18:52.597772 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.597568    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67bs6\" (UniqueName: \"kubernetes.io/projected/450dde36-b462-4b8d-a695-ff90e1fc4da9-kube-api-access-67bs6\") pod \"450dde36-b462-4b8d-a695-ff90e1fc4da9\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") "
Sep 16 05:18:52.597772 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.597741    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-tmp-dir\") pod \"450dde36-b462-4b8d-a695-ff90e1fc4da9\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") "
Sep 16 05:18:52.597861 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.597793    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-home\") pod \"450dde36-b462-4b8d-a695-ff90e1fc4da9\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") "
Sep 16 05:18:52.597861 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.597837    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-model-cache\") pod \"450dde36-b462-4b8d-a695-ff90e1fc4da9\" (UID: \"450dde36-b462-4b8d-a695-ff90e1fc4da9\") "
Sep 16 05:18:52.597999 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.597974    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "450dde36-b462-4b8d-a695-ff90e1fc4da9" (UID: "450dde36-b462-4b8d-a695-ff90e1fc4da9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:18:52.598154 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.598078    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-home" (OuterVolumeSpecName: "home") pod "450dde36-b462-4b8d-a695-ff90e1fc4da9" (UID: "450dde36-b462-4b8d-a695-ff90e1fc4da9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:18:52.598154 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.598115    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:18:52.598264 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.598166    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-model-cache" (OuterVolumeSpecName: "model-cache") pod "450dde36-b462-4b8d-a695-ff90e1fc4da9" (UID: "450dde36-b462-4b8d-a695-ff90e1fc4da9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:18:52.599671 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.599645    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-dshm" (OuterVolumeSpecName: "dshm") pod "450dde36-b462-4b8d-a695-ff90e1fc4da9" (UID: "450dde36-b462-4b8d-a695-ff90e1fc4da9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:18:52.600138 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.600116    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/450dde36-b462-4b8d-a695-ff90e1fc4da9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "450dde36-b462-4b8d-a695-ff90e1fc4da9" (UID: "450dde36-b462-4b8d-a695-ff90e1fc4da9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:18:52.600234 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.600147    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450dde36-b462-4b8d-a695-ff90e1fc4da9-kube-api-access-67bs6" (OuterVolumeSpecName: "kube-api-access-67bs6") pod "450dde36-b462-4b8d-a695-ff90e1fc4da9" (UID: "450dde36-b462-4b8d-a695-ff90e1fc4da9"). InnerVolumeSpecName "kube-api-access-67bs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:18:52.675682 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.675643    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:18:52.698939 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.698912    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:18:52.698939 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.698932    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67bs6\" (UniqueName: \"kubernetes.io/projected/450dde36-b462-4b8d-a695-ff90e1fc4da9-kube-api-access-67bs6\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:18:52.698939 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.698941    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:18:52.699146 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.698950    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450dde36-b462-4b8d-a695-ff90e1fc4da9-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:18:52.699146 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:52.698959    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450dde36-b462-4b8d-a695-ff90e1fc4da9-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:18:53.118337 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:53.118305    2573 generic.go:356] "Generic (PLEG): container finished" podID="450dde36-b462-4b8d-a695-ff90e1fc4da9" containerID="a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776" exitCode=0
Sep 16 05:18:53.118534 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:53.118343    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"
Sep 16 05:18:53.118534 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:53.118394    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k" event={"ID":"450dde36-b462-4b8d-a695-ff90e1fc4da9","Type":"ContainerDied","Data":"a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776"}
Sep 16 05:18:53.118534 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:53.118431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k" event={"ID":"450dde36-b462-4b8d-a695-ff90e1fc4da9","Type":"ContainerDied","Data":"a45bcbcb21ebd1929e5e145aebc187bd166f0484f5bef2ef861d598c7675638f"}
Sep 16 05:18:53.118534 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:53.118448    2573 scope.go:117] "RemoveContainer" containerID="a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776"
Sep 16 05:18:53.127850 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:53.127834    2573 scope.go:117] "RemoveContainer" containerID="a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776"
Sep 16 05:18:53.128157 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:18:53.128137    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776\": container with ID starting with a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776 not found: ID does not exist" containerID="a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776"
Sep 16 05:18:53.128218 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:53.128166    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776"} err="failed to get container status \"a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776\": rpc error: code = NotFound desc = could not find container \"a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776\": container with ID starting with a18f33a344e0dc0191ff23c0327f6fb7e841060836f3d68bd71a9c56bd1c1776 not found: ID does not exist"
Sep 16 05:18:53.133985 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:53.133962    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"]
Sep 16 05:18:53.137430 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:53.137406    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-10ee79/sub-test-model-10ee79-kserve-6686b98fbf-dtd5k"]
Sep 16 05:18:54.786494 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:54.786461    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="450dde36-b462-4b8d-a695-ff90e1fc4da9" path="/var/lib/kubelet/pods/450dde36-b462-4b8d-a695-ff90e1fc4da9/volumes"
Sep 16 05:18:56.840855 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:56.840829    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:18:56.843168 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:56.843146    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:18:59.847552 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:59.847507    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:59.848058 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:59.847577    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:18:59.860267 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:18:59.860240    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:19:00.165737 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:00.165705    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:19:15.783795 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:15.783749    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:19.288166 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:19.288134    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:19:35.177012 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:35.176972    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:19:44.793214 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:44.793170    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:19:51.092611 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:51.092562    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"]
Sep 16 05:19:51.123456 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:51.123417    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:52.704559 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:52.704521    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:52.887730 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:52.887699    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:55.254033 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:55.253999    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:55.574417 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:55.574325    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:19:55.930680 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:55.930643    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:56.030688 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:56.030655    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:56.137169 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:56.137129    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:56.825725 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:56.825684    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:57.606842 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.606807    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"]
Sep 16 05:19:57.607199 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.607184    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="450dde36-b462-4b8d-a695-ff90e1fc4da9" containerName="main"
Sep 16 05:19:57.607282 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.607199    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="450dde36-b462-4b8d-a695-ff90e1fc4da9" containerName="main"
Sep 16 05:19:57.607282 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.607255    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="450dde36-b462-4b8d-a695-ff90e1fc4da9" containerName="main"
Sep 16 05:19:57.610399 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.610376    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.612219 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.612197    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0653f3-serving-cert\""
Sep 16 05:19:57.619850 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.619820    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"]
Sep 16 05:19:57.636658 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.636617    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:57.677989 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.677953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5g8p\" (UniqueName: \"kubernetes.io/projected/d5558200-e08c-48e3-b8b1-815ce8dbd94a-kube-api-access-m5g8p\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.678184 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.678008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-metrics-certs\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.678184 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.678057    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5558200-e08c-48e3-b8b1-815ce8dbd94a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.678184 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.678116    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.700549 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.700523    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:57.779055 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.779018    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.779253 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.779088    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5g8p\" (UniqueName: \"kubernetes.io/projected/d5558200-e08c-48e3-b8b1-815ce8dbd94a-kube-api-access-m5g8p\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.779253 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.779129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-metrics-certs\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.779253 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.779169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5558200-e08c-48e3-b8b1-815ce8dbd94a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.779945 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.779918    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5558200-e08c-48e3-b8b1-815ce8dbd94a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.781818 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.781793    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-metrics-certs\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.781928 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.781823    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.786310 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.786284    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5g8p\" (UniqueName: \"kubernetes.io/projected/d5558200-e08c-48e3-b8b1-815ce8dbd94a-kube-api-access-m5g8p\") pod \"maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:57.803733 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.803694    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:57.924295 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:57.924255    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:58.082370 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.082336    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:58.180844 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.180761    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:19:58.188785 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.188756    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"]
Sep 16 05:19:58.194083 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.194050    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:19:58.197039 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.197013    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0653f3\""
Sep 16 05:19:58.197291 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.197083    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0653f3-dockercfg-b27mk\""
Sep 16 05:19:58.207703 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.207668    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"]
Sep 16 05:19:58.212548 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.212517    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"]
Sep 16 05:19:58.216343 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:19:58.216298    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5558200_e08c_48e3_b8b1_815ce8dbd94a.slice/crio-74e54a1e249fc9ba6aa083df8ef96cd1628017781e1a05a4121236b8f4e13a33 WatchSource:0}: Error finding container 74e54a1e249fc9ba6aa083df8ef96cd1628017781e1a05a4121236b8f4e13a33: Status 404 returned error can't find the container with id 74e54a1e249fc9ba6aa083df8ef96cd1628017781e1a05a4121236b8f4e13a33
Sep 16 05:19:58.218338 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.218319    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 05:19:58.220302 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.220268    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"]
Sep 16 05:19:58.225032 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.225011    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:19:58.235351 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.235326    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"]
Sep 16 05:19:58.284264 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.284231    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fttj7\" (UniqueName: \"kubernetes.io/projected/d9376b86-7fcd-4e03-b819-6cb7490759ae-kube-api-access-fttj7\") pod \"payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4\" (UID: \"d9376b86-7fcd-4e03-b819-6cb7490759ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:19:58.284478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.284302    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e55d26-43d1-465b-a6bb-daf7aee94a58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2\" (UID: \"48e55d26-43d1-465b-a6bb-daf7aee94a58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:19:58.284478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.284346    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9376b86-7fcd-4e03-b819-6cb7490759ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4\" (UID: \"d9376b86-7fcd-4e03-b819-6cb7490759ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:19:58.284478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.284406    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlp6b\" (UniqueName: \"kubernetes.io/projected/48e55d26-43d1-465b-a6bb-daf7aee94a58-kube-api-access-vlp6b\") pod \"payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2\" (UID: \"48e55d26-43d1-465b-a6bb-daf7aee94a58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:19:58.385301 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.385261    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fttj7\" (UniqueName: \"kubernetes.io/projected/d9376b86-7fcd-4e03-b819-6cb7490759ae-kube-api-access-fttj7\") pod \"payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4\" (UID: \"d9376b86-7fcd-4e03-b819-6cb7490759ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:19:58.385491 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.385314    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e55d26-43d1-465b-a6bb-daf7aee94a58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2\" (UID: \"48e55d26-43d1-465b-a6bb-daf7aee94a58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:19:58.385491 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.385336    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9376b86-7fcd-4e03-b819-6cb7490759ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4\" (UID: \"d9376b86-7fcd-4e03-b819-6cb7490759ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:19:58.385491 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.385377    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlp6b\" (UniqueName: \"kubernetes.io/projected/48e55d26-43d1-465b-a6bb-daf7aee94a58-kube-api-access-vlp6b\") pod \"payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2\" (UID: \"48e55d26-43d1-465b-a6bb-daf7aee94a58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:19:58.386130 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.386109    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9376b86-7fcd-4e03-b819-6cb7490759ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4\" (UID: \"d9376b86-7fcd-4e03-b819-6cb7490759ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:19:58.386176 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.386137    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e55d26-43d1-465b-a6bb-daf7aee94a58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2\" (UID: \"48e55d26-43d1-465b-a6bb-daf7aee94a58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:19:58.394045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.394014    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlp6b\" (UniqueName: \"kubernetes.io/projected/48e55d26-43d1-465b-a6bb-daf7aee94a58-kube-api-access-vlp6b\") pod \"payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2\" (UID: \"48e55d26-43d1-465b-a6bb-daf7aee94a58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:19:58.394045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.394026    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fttj7\" (UniqueName: \"kubernetes.io/projected/d9376b86-7fcd-4e03-b819-6cb7490759ae-kube-api-access-fttj7\") pod \"payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4\" (UID: \"d9376b86-7fcd-4e03-b819-6cb7490759ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:19:58.428465 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.428423    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648" event={"ID":"d5558200-e08c-48e3-b8b1-815ce8dbd94a","Type":"ContainerStarted","Data":"74e54a1e249fc9ba6aa083df8ef96cd1628017781e1a05a4121236b8f4e13a33"}
Sep 16 05:19:58.507143 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.507042    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:19:58.537721 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.537680    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:19:58.787208 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.787186    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"]
Sep 16 05:19:58.788678 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:19:58.788652    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e55d26_43d1_465b_a6bb_daf7aee94a58.slice/crio-b83b2d67f36383bf41d584362ebe2111c14ec87af1621110f4da254959481222 WatchSource:0}: Error finding container b83b2d67f36383bf41d584362ebe2111c14ec87af1621110f4da254959481222: Status 404 returned error can't find the container with id b83b2d67f36383bf41d584362ebe2111c14ec87af1621110f4da254959481222
Sep 16 05:19:58.831902 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:58.831872    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"]
Sep 16 05:19:58.836634 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:19:58.836586    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9376b86_7fcd_4e03_b819_6cb7490759ae.slice/crio-1615dbfa1899773c628f39e0177841f3877888c27b27912bc3e737cd7ffb1c6f WatchSource:0}: Error finding container 1615dbfa1899773c628f39e0177841f3877888c27b27912bc3e737cd7ffb1c6f: Status 404 returned error can't find the container with id 1615dbfa1899773c628f39e0177841f3877888c27b27912bc3e737cd7ffb1c6f
Sep 16 05:19:59.435990 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.435949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2" event={"ID":"48e55d26-43d1-465b-a6bb-daf7aee94a58","Type":"ContainerStarted","Data":"f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381"}
Sep 16 05:19:59.436483 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.436021    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:19:59.436483 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.436046    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2" event={"ID":"48e55d26-43d1-465b-a6bb-daf7aee94a58","Type":"ContainerStarted","Data":"b83b2d67f36383bf41d584362ebe2111c14ec87af1621110f4da254959481222"}
Sep 16 05:19:59.437973 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.437941    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648" event={"ID":"d5558200-e08c-48e3-b8b1-815ce8dbd94a","Type":"ContainerStarted","Data":"8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b"}
Sep 16 05:19:59.438215 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.438178    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:19:59.439811 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.439785    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4" event={"ID":"d9376b86-7fcd-4e03-b819-6cb7490759ae","Type":"ContainerStarted","Data":"4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55"}
Sep 16 05:19:59.439950 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.439830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4" event={"ID":"d9376b86-7fcd-4e03-b819-6cb7490759ae","Type":"ContainerStarted","Data":"1615dbfa1899773c628f39e0177841f3877888c27b27912bc3e737cd7ffb1c6f"}
Sep 16 05:19:59.439950 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.439918    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:19:59.451784 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.451701    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2" podStartSLOduration=1.451685419 podStartE2EDuration="1.451685419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:19:59.450392519 +0000 UTC m=+1863.116322127" watchObservedRunningTime="2026-09-16 05:19:59.451685419 +0000 UTC m=+1863.117615009"
Sep 16 05:19:59.465276 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.465211    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4" podStartSLOduration=1.46519176 podStartE2EDuration="1.46519176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:19:59.465096993 +0000 UTC m=+1863.131026613" watchObservedRunningTime="2026-09-16 05:19:59.46519176 +0000 UTC m=+1863.131121380"
Sep 16 05:19:59.479923 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:19:59.479864    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648" podStartSLOduration=2.037504398 podStartE2EDuration="2.4798389s" totalImagesPullingTime="442.334502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:19:59.479202528 +0000 UTC m=+1863.145132184" watchObservedRunningTime="2026-09-16 05:19:59.4798389 +0000 UTC m=+1863.145768493"
Sep 16 05:20:05.451270 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:05.451235    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:20:06.112247 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.112202    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6" podUID="78fdc426-e1fe-48fe-8307-df8ecfc79203" containerName="main" containerID="cri-o://e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b" gracePeriod=45
Sep 16 05:20:06.363687 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.363620    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:20:06.461758 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.461721    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-tmp-dir\") pod \"78fdc426-e1fe-48fe-8307-df8ecfc79203\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") "
Sep 16 05:20:06.461758 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.461760    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-model-cache\") pod \"78fdc426-e1fe-48fe-8307-df8ecfc79203\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") "
Sep 16 05:20:06.462297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.461800    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-home\") pod \"78fdc426-e1fe-48fe-8307-df8ecfc79203\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") "
Sep 16 05:20:06.462297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.461823    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-dshm\") pod \"78fdc426-e1fe-48fe-8307-df8ecfc79203\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") "
Sep 16 05:20:06.462297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.461874    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn4w5\" (UniqueName: \"kubernetes.io/projected/78fdc426-e1fe-48fe-8307-df8ecfc79203-kube-api-access-dn4w5\") pod \"78fdc426-e1fe-48fe-8307-df8ecfc79203\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") "
Sep 16 05:20:06.462297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.461946    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fdc426-e1fe-48fe-8307-df8ecfc79203-tls-certs\") pod \"78fdc426-e1fe-48fe-8307-df8ecfc79203\" (UID: \"78fdc426-e1fe-48fe-8307-df8ecfc79203\") "
Sep 16 05:20:06.462297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.462078    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78fdc426-e1fe-48fe-8307-df8ecfc79203" (UID: "78fdc426-e1fe-48fe-8307-df8ecfc79203"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:20:06.462297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.462110    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-home" (OuterVolumeSpecName: "home") pod "78fdc426-e1fe-48fe-8307-df8ecfc79203" (UID: "78fdc426-e1fe-48fe-8307-df8ecfc79203"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:20:06.462297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.462092    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-model-cache" (OuterVolumeSpecName: "model-cache") pod "78fdc426-e1fe-48fe-8307-df8ecfc79203" (UID: "78fdc426-e1fe-48fe-8307-df8ecfc79203"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:20:06.462297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.462232    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:20:06.462297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.462247    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:20:06.462297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.462255    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:20:06.464279 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.464252    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78fdc426-e1fe-48fe-8307-df8ecfc79203-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78fdc426-e1fe-48fe-8307-df8ecfc79203" (UID: "78fdc426-e1fe-48fe-8307-df8ecfc79203"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:20:06.464400 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.464292    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-dshm" (OuterVolumeSpecName: "dshm") pod "78fdc426-e1fe-48fe-8307-df8ecfc79203" (UID: "78fdc426-e1fe-48fe-8307-df8ecfc79203"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:20:06.464400 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.464304    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fdc426-e1fe-48fe-8307-df8ecfc79203-kube-api-access-dn4w5" (OuterVolumeSpecName: "kube-api-access-dn4w5") pod "78fdc426-e1fe-48fe-8307-df8ecfc79203" (UID: "78fdc426-e1fe-48fe-8307-df8ecfc79203"). InnerVolumeSpecName "kube-api-access-dn4w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:20:06.476739 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.476706    2573 generic.go:356] "Generic (PLEG): container finished" podID="78fdc426-e1fe-48fe-8307-df8ecfc79203" containerID="e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b" exitCode=0
Sep 16 05:20:06.476861 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.476746    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"
Sep 16 05:20:06.476861 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.476790    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6" event={"ID":"78fdc426-e1fe-48fe-8307-df8ecfc79203","Type":"ContainerDied","Data":"e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b"}
Sep 16 05:20:06.476861 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.476837    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6" event={"ID":"78fdc426-e1fe-48fe-8307-df8ecfc79203","Type":"ContainerDied","Data":"bcba237ef571659c81514e158aee38047a028c797dc40685a06b5877cccc61b5"}
Sep 16 05:20:06.476982 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.476860    2573 scope.go:117] "RemoveContainer" containerID="e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b"
Sep 16 05:20:06.480257 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.480232    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:20:06.486814 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.486789    2573 scope.go:117] "RemoveContainer" containerID="e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b"
Sep 16 05:20:06.487160 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:20:06.487135    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b\": container with ID starting with e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b not found: ID does not exist" containerID="e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b"
Sep 16 05:20:06.487272 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.487168    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b"} err="failed to get container status \"e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b\": rpc error: code = NotFound desc = could not find container \"e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b\": container with ID starting with e94bfe70bb6f1ad2cc8f4cf369de777d4bbfd73f980939141fa7511b7591d32b not found: ID does not exist"
Sep 16 05:20:06.498024 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.498001    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"]
Sep 16 05:20:06.504322 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.504292    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3784e2/rate-test-model-3784e2-kserve-5fcf6cfbbc-2f6v6"]
Sep 16 05:20:06.563812 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.563775    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fdc426-e1fe-48fe-8307-df8ecfc79203-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:20:06.563812 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.563802    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fdc426-e1fe-48fe-8307-df8ecfc79203-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:20:06.563812 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.563811    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn4w5\" (UniqueName: \"kubernetes.io/projected/78fdc426-e1fe-48fe-8307-df8ecfc79203-kube-api-access-dn4w5\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:20:06.786725 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:06.786690    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78fdc426-e1fe-48fe-8307-df8ecfc79203" path="/var/lib/kubelet/pods/78fdc426-e1fe-48fe-8307-df8ecfc79203/volumes"
Sep 16 05:20:10.447793 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:10.447760    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:20:10.448219 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:10.447841    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:20:14.909348 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:14.909314    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:19.293656 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:19.293621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:20.631269 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:20.631233    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:20.916524 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:20.916443    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:21.538624 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:21.538581    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:22.161871 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:22.161844    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:22.802125 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:22.802089    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:26.577173 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:26.577146    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:26.896561 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:26.896526    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:26.996644 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:26.996614    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:27.598886 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:27.598857    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:28.729655 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:28.729624    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:29.485659 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:29.485625    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:30.168742 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:30.168711    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:30.262740 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:30.262705    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:30.439775 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:30.439692    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:31.170104 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:31.170067    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:31.928606 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:31.928562    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:32.627732 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:32.627699    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:32.727393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:32.727360    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:32.826370 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:32.826317    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:33.072362 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.072265    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:33.170592 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.170546    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:33.177166 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.177142    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"]
Sep 16 05:20:33.177510 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.177498    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78fdc426-e1fe-48fe-8307-df8ecfc79203" containerName="main"
Sep 16 05:20:33.177554 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.177512    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="78fdc426-e1fe-48fe-8307-df8ecfc79203" containerName="main"
Sep 16 05:20:33.177589 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.177569    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="78fdc426-e1fe-48fe-8307-df8ecfc79203" containerName="main"
Sep 16 05:20:33.180340 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.180321    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:20:33.182394 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.182375    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-afd377\""
Sep 16 05:20:33.182556 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.182543    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-afd377-dockercfg-xfm8x\""
Sep 16 05:20:33.190603 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.190577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhjg\" (UniqueName: \"kubernetes.io/projected/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-kube-api-access-mfhjg\") pod \"payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g\" (UID: \"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:20:33.190686 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.190622    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g\" (UID: \"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:20:33.195846 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.195820    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"]
Sep 16 05:20:33.291520 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.291485    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhjg\" (UniqueName: \"kubernetes.io/projected/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-kube-api-access-mfhjg\") pod \"payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g\" (UID: \"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:20:33.291705 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.291542    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g\" (UID: \"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:20:33.292243 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.292221    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g\" (UID: \"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:20:33.300577 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.300551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhjg\" (UniqueName: \"kubernetes.io/projected/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-kube-api-access-mfhjg\") pod \"payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g\" (UID: \"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:20:33.489936 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.489902    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:20:33.764148 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:33.764123    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"]
Sep 16 05:20:33.766583 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:20:33.766557    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a6b5d3_da1e_4756_9d3d_3bdbccf2c426.slice/crio-7374068bc41a37095574ce81311d155cbf605511bde22788a94e8e03f82249d1 WatchSource:0}: Error finding container 7374068bc41a37095574ce81311d155cbf605511bde22788a94e8e03f82249d1: Status 404 returned error can't find the container with id 7374068bc41a37095574ce81311d155cbf605511bde22788a94e8e03f82249d1
Sep 16 05:20:34.624439 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:34.624404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g" event={"ID":"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426","Type":"ContainerStarted","Data":"ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae"}
Sep 16 05:20:34.624660 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:34.624460    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g" event={"ID":"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426","Type":"ContainerStarted","Data":"7374068bc41a37095574ce81311d155cbf605511bde22788a94e8e03f82249d1"}
Sep 16 05:20:34.624660 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:34.624561    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:20:34.643740 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:34.643687    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g" podStartSLOduration=1.6436735869999999 podStartE2EDuration="1.643673587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:20:34.641897295 +0000 UTC m=+1898.307826921" watchObservedRunningTime="2026-09-16 05:20:34.643673587 +0000 UTC m=+1898.309603180"
Sep 16 05:20:37.983982 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:37.983944    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:20:45.632778 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:45.632744    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:20:48.627935 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:48.627901    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:54.426579 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:54.426549    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:55.628483 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:55.628446    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:56.265623 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:56.265576    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:57.046931 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:57.046905    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:57.665293 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:57.665261    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:20:58.438515 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:20:58.438482    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:00.250076 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:00.250048    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:08.543188 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.543148    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"]
Sep 16 05:21:08.547213 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.547184    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.549572 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.549550    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0653f3\"/\"kube-root-ca.crt\""
Sep 16 05:21:08.549718 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.549610    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0653f3\"/\"openshift-service-ca.crt\""
Sep 16 05:21:08.550567 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.550548    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0653f3\"/\"ipp-route-0653f3-kserve-self-signed-certs\""
Sep 16 05:21:08.550700 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.550647    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0653f3\"/\"default-dockercfg-fm5pz\""
Sep 16 05:21:08.557159 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.556881    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"]
Sep 16 05:21:08.581486 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.581453    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:08.605037 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.605008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-home\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.605170 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.605055    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bb660d-35de-472a-a09d-2e651d5d8cfb-tls-certs\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.605170 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.605138    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74fw\" (UniqueName: \"kubernetes.io/projected/82bb660d-35de-472a-a09d-2e651d5d8cfb-kube-api-access-r74fw\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.605170 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.605165    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-model-cache\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.605293 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.605182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-tmp-dir\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.605293 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.605199    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-dshm\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.706359 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.706321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-home\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.706657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.706375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bb660d-35de-472a-a09d-2e651d5d8cfb-tls-certs\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.706657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.706441    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74fw\" (UniqueName: \"kubernetes.io/projected/82bb660d-35de-472a-a09d-2e651d5d8cfb-kube-api-access-r74fw\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.706657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.706479    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-model-cache\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.706657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.706503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-tmp-dir\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.706657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.706528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-dshm\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.707092 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.706807    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-home\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.707092 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.706849    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-tmp-dir\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.707092 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.706915    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-model-cache\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.708788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.708757    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-dshm\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.709080 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.709060    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bb660d-35de-472a-a09d-2e651d5d8cfb-tls-certs\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.714460 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.714439    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74fw\" (UniqueName: \"kubernetes.io/projected/82bb660d-35de-472a-a09d-2e651d5d8cfb-kube-api-access-r74fw\") pod \"ipp-route-0653f3-kserve-66c9bbfb8c-sgv26\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") " pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:08.861764 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:08.861685    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:09.137880 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:09.137835    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"]
Sep 16 05:21:09.140245 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:21:09.140212    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82bb660d_35de_472a_a09d_2e651d5d8cfb.slice/crio-363891df69c20bd8e8391d469029b44bbc7ca47d71a91079632e2d3e0d802c5c WatchSource:0}: Error finding container 363891df69c20bd8e8391d469029b44bbc7ca47d71a91079632e2d3e0d802c5c: Status 404 returned error can't find the container with id 363891df69c20bd8e8391d469029b44bbc7ca47d71a91079632e2d3e0d802c5c
Sep 16 05:21:09.791385 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:09.791349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26" event={"ID":"82bb660d-35de-472a-a09d-2e651d5d8cfb","Type":"ContainerStarted","Data":"1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172"}
Sep 16 05:21:09.791842 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:09.791397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26" event={"ID":"82bb660d-35de-472a-a09d-2e651d5d8cfb","Type":"ContainerStarted","Data":"363891df69c20bd8e8391d469029b44bbc7ca47d71a91079632e2d3e0d802c5c"}
Sep 16 05:21:09.812014 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:09.811968    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26" podStartSLOduration=1.811952811 podStartE2EDuration="1.811952811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:21:09.810728593 +0000 UTC m=+1933.476658185" watchObservedRunningTime="2026-09-16 05:21:09.811952811 +0000 UTC m=+1933.477882403"
Sep 16 05:21:12.985650 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:12.985612    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:21:16.658718 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:16.658687    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:18.862127 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:18.862092    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:18.862515 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:18.862142    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:18.874762 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:18.874732    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:19.854908 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:19.854881    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:21:29.679072 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:29.679035    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:21:36.005637 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:36.005588    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4_d9376b86-7fcd-4e03-b819-6cb7490759ae/payload-processing/0.log"
Sep 16 05:21:36.534565 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:36.534529    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:36.836438 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:36.836360    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:37.225553 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:37.225522    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:38.943345 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:38.943312    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:39.627420 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:39.627386    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:39.727301 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:39.727261    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:40.553318 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:40.553280    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:41.300053 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:41.300015    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:42.013422 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:42.013372    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:42.683186 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:42.683155    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:42.803201 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:42.803155    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:43.167215 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.167187    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:43.256572 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.256542    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:43.260237 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.260211    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"]
Sep 16 05:21:43.263610 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.263584    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:21:43.265644 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.265624    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f644d7\""
Sep 16 05:21:43.265751 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.265660    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f644d7-dockercfg-pzwbr\""
Sep 16 05:21:43.277287 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.277267    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"]
Sep 16 05:21:43.294195 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.294163    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"]
Sep 16 05:21:43.297582 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.297564    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:21:43.310713 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.310685    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"]
Sep 16 05:21:43.359068 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.359034    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:21:43.415868 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.415838    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6\" (UID: \"5db3fde5-25b6-4c16-bf77-3ac29075c8ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:21:43.416043 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.415884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dlgl\" (UniqueName: \"kubernetes.io/projected/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-kube-api-access-9dlgl\") pod \"payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6\" (UID: \"5db3fde5-25b6-4c16-bf77-3ac29075c8ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:21:43.416043 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.415929    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4ml\" (UniqueName: \"kubernetes.io/projected/316f93c1-abea-499e-851f-029a7cd16852-kube-api-access-4z4ml\") pod \"payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg\" (UID: \"316f93c1-abea-499e-851f-029a7cd16852\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:21:43.416043 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.415963    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316f93c1-abea-499e-851f-029a7cd16852-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg\" (UID: \"316f93c1-abea-499e-851f-029a7cd16852\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:21:43.516961 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.516878    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6\" (UID: \"5db3fde5-25b6-4c16-bf77-3ac29075c8ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:21:43.516961 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.516920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dlgl\" (UniqueName: \"kubernetes.io/projected/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-kube-api-access-9dlgl\") pod \"payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6\" (UID: \"5db3fde5-25b6-4c16-bf77-3ac29075c8ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:21:43.516961 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.516952    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4ml\" (UniqueName: \"kubernetes.io/projected/316f93c1-abea-499e-851f-029a7cd16852-kube-api-access-4z4ml\") pod \"payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg\" (UID: \"316f93c1-abea-499e-851f-029a7cd16852\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:21:43.517192 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.516976    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316f93c1-abea-499e-851f-029a7cd16852-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg\" (UID: \"316f93c1-abea-499e-851f-029a7cd16852\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:21:43.517709 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.517688    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316f93c1-abea-499e-851f-029a7cd16852-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg\" (UID: \"316f93c1-abea-499e-851f-029a7cd16852\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:21:43.517788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.517711    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6\" (UID: \"5db3fde5-25b6-4c16-bf77-3ac29075c8ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:21:43.524803 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.524779    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4ml\" (UniqueName: \"kubernetes.io/projected/316f93c1-abea-499e-851f-029a7cd16852-kube-api-access-4z4ml\") pod \"payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg\" (UID: \"316f93c1-abea-499e-851f-029a7cd16852\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:21:43.524947 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.524923    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dlgl\" (UniqueName: \"kubernetes.io/projected/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-kube-api-access-9dlgl\") pod \"payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6\" (UID: \"5db3fde5-25b6-4c16-bf77-3ac29075c8ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:21:43.574364 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.574338    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:21:43.607544 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.607516    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:21:43.850463 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.850388    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"]
Sep 16 05:21:43.853913 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:21:43.853884    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db3fde5_25b6_4c16_bf77_3ac29075c8ee.slice/crio-7f1448f2347b4f1416fa649bbc98544ba0e26f619d9befcf128d6b215faf2d14 WatchSource:0}: Error finding container 7f1448f2347b4f1416fa649bbc98544ba0e26f619d9befcf128d6b215faf2d14: Status 404 returned error can't find the container with id 7f1448f2347b4f1416fa649bbc98544ba0e26f619d9befcf128d6b215faf2d14
Sep 16 05:21:43.889498 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.889473    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"]
Sep 16 05:21:43.892167 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:21:43.892107    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316f93c1_abea_499e_851f_029a7cd16852.slice/crio-3eb4001782101631ed3e971c1d9928815dfb8a9501bafe21c109bb8f014de0ff WatchSource:0}: Error finding container 3eb4001782101631ed3e971c1d9928815dfb8a9501bafe21c109bb8f014de0ff: Status 404 returned error can't find the container with id 3eb4001782101631ed3e971c1d9928815dfb8a9501bafe21c109bb8f014de0ff
Sep 16 05:21:43.961664 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.961625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg" event={"ID":"316f93c1-abea-499e-851f-029a7cd16852","Type":"ContainerStarted","Data":"33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238"}
Sep 16 05:21:43.961816 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.961696    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg" event={"ID":"316f93c1-abea-499e-851f-029a7cd16852","Type":"ContainerStarted","Data":"3eb4001782101631ed3e971c1d9928815dfb8a9501bafe21c109bb8f014de0ff"}
Sep 16 05:21:43.961892 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.961829    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:21:43.963640 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.963582    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6" event={"ID":"5db3fde5-25b6-4c16-bf77-3ac29075c8ee","Type":"ContainerStarted","Data":"f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006"}
Sep 16 05:21:43.963797 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.963655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6" event={"ID":"5db3fde5-25b6-4c16-bf77-3ac29075c8ee","Type":"ContainerStarted","Data":"7f1448f2347b4f1416fa649bbc98544ba0e26f619d9befcf128d6b215faf2d14"}
Sep 16 05:21:43.963907 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.963886    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:21:43.977899 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.977841    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg" podStartSLOduration=0.97782063 podStartE2EDuration="977.82063ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:21:43.975899995 +0000 UTC m=+1967.641829615" watchObservedRunningTime="2026-09-16 05:21:43.97782063 +0000 UTC m=+1967.643750224"
Sep 16 05:21:43.991937 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:43.991891    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6" podStartSLOduration=0.991876571 podStartE2EDuration="991.876571ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:21:43.989789018 +0000 UTC m=+1967.655718630" watchObservedRunningTime="2026-09-16 05:21:43.991876571 +0000 UTC m=+1967.657806164"
Sep 16 05:21:49.886670 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:49.883000    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:21:54.970749 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:54.970719    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:21:54.971166 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:21:54.970794    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:22:04.558113 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:04.558069    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:04.660633 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:04.660585    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:05.317864 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:05.317829    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:05.973455 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:05.973427    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:06.649225 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:06.649189    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:07.524165 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:07.524136    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:08.134343 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:08.134306    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:08.830139 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:08.830104    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:09.642523 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:09.642487    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:11.143658 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:11.143590    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:11.720036 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:11.720004    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:11.758882 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:11.758850    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"]
Sep 16 05:22:11.759130 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:11.759104    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg" podUID="316f93c1-abea-499e-851f-029a7cd16852" containerName="payload-processing" containerID="cri-o://33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238" gracePeriod=30
Sep 16 05:22:11.768451 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:11.767029    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"]
Sep 16 05:22:11.768451 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:11.767270    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6" podUID="5db3fde5-25b6-4c16-bf77-3ac29075c8ee" containerName="payload-pre-processing" containerID="cri-o://f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006" gracePeriod=30
Sep 16 05:22:11.802247 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:11.802211    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:11.868169 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:11.868121    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:11.967004 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:11.966969    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:12.029104 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.029055    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:12.059004 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.058977    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:22:12.062245 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.062217    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:22:12.101275 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.101242    2573 generic.go:356] "Generic (PLEG): container finished" podID="5db3fde5-25b6-4c16-bf77-3ac29075c8ee" containerID="f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006" exitCode=0
Sep 16 05:22:12.101275 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.101276    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"
Sep 16 05:22:12.101526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.101323    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6" event={"ID":"5db3fde5-25b6-4c16-bf77-3ac29075c8ee","Type":"ContainerDied","Data":"f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006"}
Sep 16 05:22:12.101526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.101411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6" event={"ID":"5db3fde5-25b6-4c16-bf77-3ac29075c8ee","Type":"ContainerDied","Data":"7f1448f2347b4f1416fa649bbc98544ba0e26f619d9befcf128d6b215faf2d14"}
Sep 16 05:22:12.101526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.101442    2573 scope.go:117] "RemoveContainer" containerID="f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006"
Sep 16 05:22:12.103857 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.103820    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"
Sep 16 05:22:12.103984 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.103866    2573 generic.go:356] "Generic (PLEG): container finished" podID="316f93c1-abea-499e-851f-029a7cd16852" containerID="33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238" exitCode=0
Sep 16 05:22:12.103984 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.103940    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg" event={"ID":"316f93c1-abea-499e-851f-029a7cd16852","Type":"ContainerDied","Data":"33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238"}
Sep 16 05:22:12.104100 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.103985    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg" event={"ID":"316f93c1-abea-499e-851f-029a7cd16852","Type":"ContainerDied","Data":"3eb4001782101631ed3e971c1d9928815dfb8a9501bafe21c109bb8f014de0ff"}
Sep 16 05:22:12.111013 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.110995    2573 scope.go:117] "RemoveContainer" containerID="f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006"
Sep 16 05:22:12.111272 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:22:12.111249    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006\": container with ID starting with f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006 not found: ID does not exist" containerID="f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006"
Sep 16 05:22:12.111348 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.111282    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006"} err="failed to get container status \"f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006\": rpc error: code = NotFound desc = could not find container \"f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006\": container with ID starting with f9d0d951b8105f483abbe2d63dc3084ef276daa4f554458da1e1c46d6ccc6006 not found: ID does not exist"
Sep 16 05:22:12.111348 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.111306    2573 scope.go:117] "RemoveContainer" containerID="33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238"
Sep 16 05:22:12.121167 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.121146    2573 scope.go:117] "RemoveContainer" containerID="33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238"
Sep 16 05:22:12.121465 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:22:12.121439    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238\": container with ID starting with 33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238 not found: ID does not exist" containerID="33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238"
Sep 16 05:22:12.121517 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.121473    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238"} err="failed to get container status \"33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238\": rpc error: code = NotFound desc = could not find container \"33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238\": container with ID starting with 33c0b1130cb59797eb68af391dc1222195787167a9cb661d1d8a36dddd954238 not found: ID does not exist"
Sep 16 05:22:12.160717 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.160687    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z4ml\" (UniqueName: \"kubernetes.io/projected/316f93c1-abea-499e-851f-029a7cd16852-kube-api-access-4z4ml\") pod \"316f93c1-abea-499e-851f-029a7cd16852\" (UID: \"316f93c1-abea-499e-851f-029a7cd16852\") "
Sep 16 05:22:12.161196 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.160727    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dlgl\" (UniqueName: \"kubernetes.io/projected/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-kube-api-access-9dlgl\") pod \"5db3fde5-25b6-4c16-bf77-3ac29075c8ee\" (UID: \"5db3fde5-25b6-4c16-bf77-3ac29075c8ee\") "
Sep 16 05:22:12.161196 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.160792    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316f93c1-abea-499e-851f-029a7cd16852-plugins-config-volume\") pod \"316f93c1-abea-499e-851f-029a7cd16852\" (UID: \"316f93c1-abea-499e-851f-029a7cd16852\") "
Sep 16 05:22:12.161196 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.160852    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-plugins-config-volume\") pod \"5db3fde5-25b6-4c16-bf77-3ac29075c8ee\" (UID: \"5db3fde5-25b6-4c16-bf77-3ac29075c8ee\") "
Sep 16 05:22:12.161321 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.161263    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316f93c1-abea-499e-851f-029a7cd16852-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "316f93c1-abea-499e-851f-029a7cd16852" (UID: "316f93c1-abea-499e-851f-029a7cd16852"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:22:12.161321 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.161282    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5db3fde5-25b6-4c16-bf77-3ac29075c8ee" (UID: "5db3fde5-25b6-4c16-bf77-3ac29075c8ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:22:12.163022 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.162996    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-kube-api-access-9dlgl" (OuterVolumeSpecName: "kube-api-access-9dlgl") pod "5db3fde5-25b6-4c16-bf77-3ac29075c8ee" (UID: "5db3fde5-25b6-4c16-bf77-3ac29075c8ee"). InnerVolumeSpecName "kube-api-access-9dlgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:22:12.163139 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.163038    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316f93c1-abea-499e-851f-029a7cd16852-kube-api-access-4z4ml" (OuterVolumeSpecName: "kube-api-access-4z4ml") pod "316f93c1-abea-499e-851f-029a7cd16852" (UID: "316f93c1-abea-499e-851f-029a7cd16852"). InnerVolumeSpecName "kube-api-access-4z4ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:22:12.262488 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.262392    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316f93c1-abea-499e-851f-029a7cd16852-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:12.262488 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.262426    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:12.262488 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.262444    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z4ml\" (UniqueName: \"kubernetes.io/projected/316f93c1-abea-499e-851f-029a7cd16852-kube-api-access-4z4ml\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:12.262488 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.262452    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dlgl\" (UniqueName: \"kubernetes.io/projected/5db3fde5-25b6-4c16-bf77-3ac29075c8ee-kube-api-access-9dlgl\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:12.345464 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.345430    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:12.425124 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.425084    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"]
Sep 16 05:22:12.428376 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.428316    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:12.428957 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.428934    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f644d7-f85b4f9fd-gw2w6"]
Sep 16 05:22:12.440300 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.440254    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"]
Sep 16 05:22:12.442055 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.442034    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f644d7-8494dc7f6f-f6grg"]
Sep 16 05:22:12.787010 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.786977    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="316f93c1-abea-499e-851f-029a7cd16852" path="/var/lib/kubelet/pods/316f93c1-abea-499e-851f-029a7cd16852/volumes"
Sep 16 05:22:12.787382 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:12.787368    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db3fde5-25b6-4c16-bf77-3ac29075c8ee" path="/var/lib/kubelet/pods/5db3fde5-25b6-4c16-bf77-3ac29075c8ee/volumes"
Sep 16 05:22:13.106235 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:13.106142    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:13.777181 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:13.777153    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:14.537787 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:14.537756    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:15.161791 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:15.161750    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:16.385749 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:16.385705    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:22:18.193796 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:18.193764    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:18.295200 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:18.295157    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:18.454070 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:18.453985    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:22.238717 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:22.238682    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:23.236000 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:23.235959    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:23.915157 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:23.915124    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:24.630367 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:24.630336    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:25.355359 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:25.355319    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:26.002121 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:26.002092    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:35.281004 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:35.280950    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:22:37.466511 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.466475    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"]
Sep 16 05:22:37.466991 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.466732    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648" podUID="d5558200-e08c-48e3-b8b1-815ce8dbd94a" containerName="maas-api" containerID="cri-o://8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b" gracePeriod=30
Sep 16 05:22:37.493740 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.493700    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:37.511279 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.511240    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"]
Sep 16 05:22:37.511521 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.511496    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4" podUID="d9376b86-7fcd-4e03-b819-6cb7490759ae" containerName="payload-processing" containerID="cri-o://4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55" gracePeriod=30
Sep 16 05:22:37.522239 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.522166    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"]
Sep 16 05:22:37.522484 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.522415    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2" podUID="48e55d26-43d1-465b-a6bb-daf7aee94a58" containerName="payload-pre-processing" containerID="cri-o://f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381" gracePeriod=30
Sep 16 05:22:37.593642 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.593561    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:37.668467 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.668428    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:37.733369 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.733334    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:37.811447 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.811414    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:22:37.814810 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.814784    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:22:37.820437 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.820408    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:22:37.900949 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.900909    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fttj7\" (UniqueName: \"kubernetes.io/projected/d9376b86-7fcd-4e03-b819-6cb7490759ae-kube-api-access-fttj7\") pod \"d9376b86-7fcd-4e03-b819-6cb7490759ae\" (UID: \"d9376b86-7fcd-4e03-b819-6cb7490759ae\") "
Sep 16 05:22:37.901141 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.900965    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5558200-e08c-48e3-b8b1-815ce8dbd94a-openshift-service-ca\") pod \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") "
Sep 16 05:22:37.901141 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.900983    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9376b86-7fcd-4e03-b819-6cb7490759ae-plugins-config-volume\") pod \"d9376b86-7fcd-4e03-b819-6cb7490759ae\" (UID: \"d9376b86-7fcd-4e03-b819-6cb7490759ae\") "
Sep 16 05:22:37.901141 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.901011    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-metrics-certs\") pod \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") "
Sep 16 05:22:37.901141 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.901064    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e55d26-43d1-465b-a6bb-daf7aee94a58-plugins-config-volume\") pod \"48e55d26-43d1-465b-a6bb-daf7aee94a58\" (UID: \"48e55d26-43d1-465b-a6bb-daf7aee94a58\") "
Sep 16 05:22:37.901141 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.901103    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-maas-api-tls\") pod \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") "
Sep 16 05:22:37.901497 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.901148    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlp6b\" (UniqueName: \"kubernetes.io/projected/48e55d26-43d1-465b-a6bb-daf7aee94a58-kube-api-access-vlp6b\") pod \"48e55d26-43d1-465b-a6bb-daf7aee94a58\" (UID: \"48e55d26-43d1-465b-a6bb-daf7aee94a58\") "
Sep 16 05:22:37.901497 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.901182    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5g8p\" (UniqueName: \"kubernetes.io/projected/d5558200-e08c-48e3-b8b1-815ce8dbd94a-kube-api-access-m5g8p\") pod \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\" (UID: \"d5558200-e08c-48e3-b8b1-815ce8dbd94a\") "
Sep 16 05:22:37.901497 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.901419    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9376b86-7fcd-4e03-b819-6cb7490759ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9376b86-7fcd-4e03-b819-6cb7490759ae" (UID: "d9376b86-7fcd-4e03-b819-6cb7490759ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:22:37.901497 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.901465    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5558200-e08c-48e3-b8b1-815ce8dbd94a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5558200-e08c-48e3-b8b1-815ce8dbd94a" (UID: "d5558200-e08c-48e3-b8b1-815ce8dbd94a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:22:37.901754 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.901500    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e55d26-43d1-465b-a6bb-daf7aee94a58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "48e55d26-43d1-465b-a6bb-daf7aee94a58" (UID: "48e55d26-43d1-465b-a6bb-daf7aee94a58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:22:37.901754 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.901539    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5558200-e08c-48e3-b8b1-815ce8dbd94a-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:37.901754 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.901562    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9376b86-7fcd-4e03-b819-6cb7490759ae-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:37.903471 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.903446    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5558200-e08c-48e3-b8b1-815ce8dbd94a-kube-api-access-m5g8p" (OuterVolumeSpecName: "kube-api-access-m5g8p") pod "d5558200-e08c-48e3-b8b1-815ce8dbd94a" (UID: "d5558200-e08c-48e3-b8b1-815ce8dbd94a"). InnerVolumeSpecName "kube-api-access-m5g8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:22:37.903895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.903828    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5558200-e08c-48e3-b8b1-815ce8dbd94a" (UID: "d5558200-e08c-48e3-b8b1-815ce8dbd94a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:22:37.903895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.903861    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5558200-e08c-48e3-b8b1-815ce8dbd94a" (UID: "d5558200-e08c-48e3-b8b1-815ce8dbd94a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:22:37.903895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.903886    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9376b86-7fcd-4e03-b819-6cb7490759ae-kube-api-access-fttj7" (OuterVolumeSpecName: "kube-api-access-fttj7") pod "d9376b86-7fcd-4e03-b819-6cb7490759ae" (UID: "d9376b86-7fcd-4e03-b819-6cb7490759ae"). InnerVolumeSpecName "kube-api-access-fttj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:22:37.904060 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:37.903920    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e55d26-43d1-465b-a6bb-daf7aee94a58-kube-api-access-vlp6b" (OuterVolumeSpecName: "kube-api-access-vlp6b") pod "48e55d26-43d1-465b-a6bb-daf7aee94a58" (UID: "48e55d26-43d1-465b-a6bb-daf7aee94a58"). InnerVolumeSpecName "kube-api-access-vlp6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:22:38.003119 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.003027    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5g8p\" (UniqueName: \"kubernetes.io/projected/d5558200-e08c-48e3-b8b1-815ce8dbd94a-kube-api-access-m5g8p\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:38.003119 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.003058    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fttj7\" (UniqueName: \"kubernetes.io/projected/d9376b86-7fcd-4e03-b819-6cb7490759ae-kube-api-access-fttj7\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:38.003119 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.003068    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:38.003119 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.003077    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e55d26-43d1-465b-a6bb-daf7aee94a58-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:38.003119 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.003088    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5558200-e08c-48e3-b8b1-815ce8dbd94a-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:38.003119 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.003099    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlp6b\" (UniqueName: \"kubernetes.io/projected/48e55d26-43d1-465b-a6bb-daf7aee94a58-kube-api-access-vlp6b\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:38.051747 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.051695    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:38.157327 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.157280    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:38.242936 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.242900    2573 generic.go:356] "Generic (PLEG): container finished" podID="d9376b86-7fcd-4e03-b819-6cb7490759ae" containerID="4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55" exitCode=0
Sep 16 05:22:38.243150 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.242949    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"
Sep 16 05:22:38.243150 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.242989    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4" event={"ID":"d9376b86-7fcd-4e03-b819-6cb7490759ae","Type":"ContainerDied","Data":"4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55"}
Sep 16 05:22:38.243150 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.243054    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4" event={"ID":"d9376b86-7fcd-4e03-b819-6cb7490759ae","Type":"ContainerDied","Data":"1615dbfa1899773c628f39e0177841f3877888c27b27912bc3e737cd7ffb1c6f"}
Sep 16 05:22:38.243150 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.243080    2573 scope.go:117] "RemoveContainer" containerID="4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55"
Sep 16 05:22:38.244452 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.244432    2573 generic.go:356] "Generic (PLEG): container finished" podID="48e55d26-43d1-465b-a6bb-daf7aee94a58" containerID="f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381" exitCode=0
Sep 16 05:22:38.244523 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.244489    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"
Sep 16 05:22:38.244523 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.244494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2" event={"ID":"48e55d26-43d1-465b-a6bb-daf7aee94a58","Type":"ContainerDied","Data":"f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381"}
Sep 16 05:22:38.244654 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.244558    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2" event={"ID":"48e55d26-43d1-465b-a6bb-daf7aee94a58","Type":"ContainerDied","Data":"b83b2d67f36383bf41d584362ebe2111c14ec87af1621110f4da254959481222"}
Sep 16 05:22:38.246085 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.246048    2573 generic.go:356] "Generic (PLEG): container finished" podID="d5558200-e08c-48e3-b8b1-815ce8dbd94a" containerID="8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b" exitCode=0
Sep 16 05:22:38.246228 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.246087    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"
Sep 16 05:22:38.246228 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.246130    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648" event={"ID":"d5558200-e08c-48e3-b8b1-815ce8dbd94a","Type":"ContainerDied","Data":"8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b"}
Sep 16 05:22:38.246228 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.246188    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648" event={"ID":"d5558200-e08c-48e3-b8b1-815ce8dbd94a","Type":"ContainerDied","Data":"74e54a1e249fc9ba6aa083df8ef96cd1628017781e1a05a4121236b8f4e13a33"}
Sep 16 05:22:38.255183 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.255159    2573 scope.go:117] "RemoveContainer" containerID="4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55"
Sep 16 05:22:38.255536 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:22:38.255514    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55\": container with ID starting with 4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55 not found: ID does not exist" containerID="4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55"
Sep 16 05:22:38.255663 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.255547    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55"} err="failed to get container status \"4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55\": rpc error: code = NotFound desc = could not find container \"4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55\": container with ID starting with 4df43f7a8261773df91de660646f0466f3d08afd7d4b030c0dac58429da0fb55 not found: ID does not exist"
Sep 16 05:22:38.255663 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.255573    2573 scope.go:117] "RemoveContainer" containerID="f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381"
Sep 16 05:22:38.264710 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.264686    2573 scope.go:117] "RemoveContainer" containerID="f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381"
Sep 16 05:22:38.265054 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:22:38.265035    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381\": container with ID starting with f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381 not found: ID does not exist" containerID="f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381"
Sep 16 05:22:38.265102 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.265061    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381"} err="failed to get container status \"f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381\": rpc error: code = NotFound desc = could not find container \"f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381\": container with ID starting with f1fb0c5a57697f74a8266c83899d06e8bc24b8a8ceac967a652160bce4382381 not found: ID does not exist"
Sep 16 05:22:38.265102 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.265078    2573 scope.go:117] "RemoveContainer" containerID="8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b"
Sep 16 05:22:38.274427 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.274404    2573 scope.go:117] "RemoveContainer" containerID="8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b"
Sep 16 05:22:38.274807 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:22:38.274785    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b\": container with ID starting with 8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b not found: ID does not exist" containerID="8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b"
Sep 16 05:22:38.274883 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.274814    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b"} err="failed to get container status \"8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b\": rpc error: code = NotFound desc = could not find container \"8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b\": container with ID starting with 8f208ead4f2cee639b7dcbcaa4afe5899326c311ce449404bc0d1fe5ea9f398b not found: ID does not exist"
Sep 16 05:22:38.279837 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.279799    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"]
Sep 16 05:22:38.296880 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.296828    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0653f3-5bc66dfd86-n4dj2"]
Sep 16 05:22:38.311156 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.311121    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"]
Sep 16 05:22:38.312873 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.312849    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0653f3-747f6f7cf9-24pp4"]
Sep 16 05:22:38.326850 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.326508    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"]
Sep 16 05:22:38.333798 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.333764    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0653f3-5f795d6468-dm648"]
Sep 16 05:22:38.787159 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.787122    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48e55d26-43d1-465b-a6bb-daf7aee94a58" path="/var/lib/kubelet/pods/48e55d26-43d1-465b-a6bb-daf7aee94a58/volumes"
Sep 16 05:22:38.787551 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.787533    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5558200-e08c-48e3-b8b1-815ce8dbd94a" path="/var/lib/kubelet/pods/d5558200-e08c-48e3-b8b1-815ce8dbd94a/volumes"
Sep 16 05:22:38.787883 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.787869    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9376b86-7fcd-4e03-b819-6cb7490759ae" path="/var/lib/kubelet/pods/d9376b86-7fcd-4e03-b819-6cb7490759ae/volumes"
Sep 16 05:22:38.842147 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:38.842114    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:39.831353 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:39.831267    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:41.235166 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:41.235128    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:43.028566 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:43.028532    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"]
Sep 16 05:22:43.125365 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:43.125324    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:47.980178 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:47.980141    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:22:56.772021 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:56.771967    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:22:58.295435 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.295372    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26" podUID="82bb660d-35de-472a-a09d-2e651d5d8cfb" containerName="main" containerID="cri-o://1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172" gracePeriod=45
Sep 16 05:22:58.557144 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.557074    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:22:58.583851 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.583814    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r74fw\" (UniqueName: \"kubernetes.io/projected/82bb660d-35de-472a-a09d-2e651d5d8cfb-kube-api-access-r74fw\") pod \"82bb660d-35de-472a-a09d-2e651d5d8cfb\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") "
Sep 16 05:22:58.584045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.583899    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-model-cache\") pod \"82bb660d-35de-472a-a09d-2e651d5d8cfb\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") "
Sep 16 05:22:58.584045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.583929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-home\") pod \"82bb660d-35de-472a-a09d-2e651d5d8cfb\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") "
Sep 16 05:22:58.584045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.583953    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bb660d-35de-472a-a09d-2e651d5d8cfb-tls-certs\") pod \"82bb660d-35de-472a-a09d-2e651d5d8cfb\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") "
Sep 16 05:22:58.584045 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.584025    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-tmp-dir\") pod \"82bb660d-35de-472a-a09d-2e651d5d8cfb\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") "
Sep 16 05:22:58.584264 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.584069    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-dshm\") pod \"82bb660d-35de-472a-a09d-2e651d5d8cfb\" (UID: \"82bb660d-35de-472a-a09d-2e651d5d8cfb\") "
Sep 16 05:22:58.584264 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.584167    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-model-cache" (OuterVolumeSpecName: "model-cache") pod "82bb660d-35de-472a-a09d-2e651d5d8cfb" (UID: "82bb660d-35de-472a-a09d-2e651d5d8cfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:22:58.584393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.584339    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:58.584393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.584383    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82bb660d-35de-472a-a09d-2e651d5d8cfb" (UID: "82bb660d-35de-472a-a09d-2e651d5d8cfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:22:58.584772 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.584745    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-home" (OuterVolumeSpecName: "home") pod "82bb660d-35de-472a-a09d-2e651d5d8cfb" (UID: "82bb660d-35de-472a-a09d-2e651d5d8cfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:22:58.586536 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.586491    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-dshm" (OuterVolumeSpecName: "dshm") pod "82bb660d-35de-472a-a09d-2e651d5d8cfb" (UID: "82bb660d-35de-472a-a09d-2e651d5d8cfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:22:58.586536 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.586521    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bb660d-35de-472a-a09d-2e651d5d8cfb-kube-api-access-r74fw" (OuterVolumeSpecName: "kube-api-access-r74fw") pod "82bb660d-35de-472a-a09d-2e651d5d8cfb" (UID: "82bb660d-35de-472a-a09d-2e651d5d8cfb"). InnerVolumeSpecName "kube-api-access-r74fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:22:58.586712 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.586614    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bb660d-35de-472a-a09d-2e651d5d8cfb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82bb660d-35de-472a-a09d-2e651d5d8cfb" (UID: "82bb660d-35de-472a-a09d-2e651d5d8cfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:22:58.685704 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.685656    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:58.685704 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.685694    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bb660d-35de-472a-a09d-2e651d5d8cfb-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:58.685704 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.685708    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:58.685963 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.685718    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bb660d-35de-472a-a09d-2e651d5d8cfb-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:58.685963 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:58.685731    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r74fw\" (UniqueName: \"kubernetes.io/projected/82bb660d-35de-472a-a09d-2e651d5d8cfb-kube-api-access-r74fw\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:22:59.356542 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:59.356499    2573 generic.go:356] "Generic (PLEG): container finished" podID="82bb660d-35de-472a-a09d-2e651d5d8cfb" containerID="1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172" exitCode=0
Sep 16 05:22:59.356542 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:59.356546    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"
Sep 16 05:22:59.357098 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:59.356582    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26" event={"ID":"82bb660d-35de-472a-a09d-2e651d5d8cfb","Type":"ContainerDied","Data":"1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172"}
Sep 16 05:22:59.357098 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:59.356649    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26" event={"ID":"82bb660d-35de-472a-a09d-2e651d5d8cfb","Type":"ContainerDied","Data":"363891df69c20bd8e8391d469029b44bbc7ca47d71a91079632e2d3e0d802c5c"}
Sep 16 05:22:59.357098 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:59.356666    2573 scope.go:117] "RemoveContainer" containerID="1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172"
Sep 16 05:22:59.365892 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:59.365792    2573 scope.go:117] "RemoveContainer" containerID="1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172"
Sep 16 05:22:59.366194 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:22:59.366156    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172\": container with ID starting with 1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172 not found: ID does not exist" containerID="1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172"
Sep 16 05:22:59.366320 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:59.366201    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172"} err="failed to get container status \"1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172\": rpc error: code = NotFound desc = could not find container \"1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172\": container with ID starting with 1557949aa331ebaea63d4f79b32b9b89bb56fa7f3f7907d30b7dc1c2af2bc172 not found: ID does not exist"
Sep 16 05:22:59.373583 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:59.373549    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"]
Sep 16 05:22:59.377367 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:22:59.377338    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0653f3/ipp-route-0653f3-kserve-66c9bbfb8c-sgv26"]
Sep 16 05:23:00.788360 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:00.788319    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82bb660d-35de-472a-a09d-2e651d5d8cfb" path="/var/lib/kubelet/pods/82bb660d-35de-472a-a09d-2e651d5d8cfb/volumes"
Sep 16 05:23:19.167993 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:19.167958    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:20.524070 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:20.524031    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:26.308374 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:26.308302    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:26.409364 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:26.409324    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:26.591308 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:26.591213    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:27.241522 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:27.241486    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:28.044519 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:28.044480    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:28.665830 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:28.665787    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:29.302289 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:29.302254    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:30.125819 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:30.125789    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:37.421723 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.421683    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:37.455969 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.455928    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"]
Sep 16 05:23:37.456215 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.456190    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g" podUID="07a6b5d3-da1e-4756-9d3d-3bdbccf2c426" containerName="payload-pre-processing" containerID="cri-o://ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae" gracePeriod=30
Sep 16 05:23:37.526436 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.526401    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:37.621627 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.621586    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:37.689708 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.689661    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:37.703901 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.703878    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:23:37.727743 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.727705    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfhjg\" (UniqueName: \"kubernetes.io/projected/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-kube-api-access-mfhjg\") pod \"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426\" (UID: \"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426\") "
Sep 16 05:23:37.727896 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.727809    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-plugins-config-volume\") pod \"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426\" (UID: \"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426\") "
Sep 16 05:23:37.728341 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.728305    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07a6b5d3-da1e-4756-9d3d-3bdbccf2c426" (UID: "07a6b5d3-da1e-4756-9d3d-3bdbccf2c426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:23:37.729922 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.729892    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-kube-api-access-mfhjg" (OuterVolumeSpecName: "kube-api-access-mfhjg") pod "07a6b5d3-da1e-4756-9d3d-3bdbccf2c426" (UID: "07a6b5d3-da1e-4756-9d3d-3bdbccf2c426"). InnerVolumeSpecName "kube-api-access-mfhjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:23:37.829306 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.829260    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:23:37.829306 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.829294    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfhjg\" (UniqueName: \"kubernetes.io/projected/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426-kube-api-access-mfhjg\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:23:37.979953 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:37.979855    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:38.080044 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.080001    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:38.544444 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.544394    2573 generic.go:356] "Generic (PLEG): container finished" podID="07a6b5d3-da1e-4756-9d3d-3bdbccf2c426" containerID="ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae" exitCode=0
Sep 16 05:23:38.544444 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.544444    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"
Sep 16 05:23:38.544992 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.544466    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g" event={"ID":"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426","Type":"ContainerDied","Data":"ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae"}
Sep 16 05:23:38.544992 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.544527    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g" event={"ID":"07a6b5d3-da1e-4756-9d3d-3bdbccf2c426","Type":"ContainerDied","Data":"7374068bc41a37095574ce81311d155cbf605511bde22788a94e8e03f82249d1"}
Sep 16 05:23:38.544992 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.544560    2573 scope.go:117] "RemoveContainer" containerID="ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae"
Sep 16 05:23:38.554105 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.554083    2573 scope.go:117] "RemoveContainer" containerID="ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae"
Sep 16 05:23:38.554490 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:23:38.554457    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae\": container with ID starting with ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae not found: ID does not exist" containerID="ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae"
Sep 16 05:23:38.554590 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.554504    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae"} err="failed to get container status \"ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae\": rpc error: code = NotFound desc = could not find container \"ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae\": container with ID starting with ee84f366416f5490c37d9d87651f02592e495287c4a5d8828c7d5faba8af41ae not found: ID does not exist"
Sep 16 05:23:38.566533 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.566481    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"]
Sep 16 05:23:38.575586 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.575558    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd377-68fbdd68cd-bsk9g"]
Sep 16 05:23:38.786494 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.786455    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07a6b5d3-da1e-4756-9d3d-3bdbccf2c426" path="/var/lib/kubelet/pods/07a6b5d3-da1e-4756-9d3d-3bdbccf2c426/volumes"
Sep 16 05:23:38.844232 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:38.844126    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:39.497264 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:39.497225    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:40.675899 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:40.675863    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:23:43.997472 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:43.997441    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:44.098076 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:44.098041    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:44.273303 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:44.273217    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:45.369998 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:45.369952    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:46.117437 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:46.117407    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:46.857523 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:46.857486    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:52.388943 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:52.388905    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:55.037940 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.037900    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"]
Sep 16 05:23:55.038423 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.038135    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47" podUID="9ef0e388-5c57-4d87-a2f0-d8d0d70f2137" containerName="maas-api" containerID="cri-o://60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509" gracePeriod=30
Sep 16 05:23:55.068369 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.068339    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:55.149658 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.149621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:55.222209 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.222155    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:55.282755 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.282727    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:55.290516 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.290459    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:23:55.376668 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.376638    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-metrics-certs\") pod \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") "
Sep 16 05:23:55.376861 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.376699    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcdbq\" (UniqueName: \"kubernetes.io/projected/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-kube-api-access-bcdbq\") pod \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") "
Sep 16 05:23:55.376861 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.376733    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-openshift-service-ca\") pod \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") "
Sep 16 05:23:55.376861 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.376766    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-maas-api-tls\") pod \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\" (UID: \"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137\") "
Sep 16 05:23:55.377136 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.377112    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ef0e388-5c57-4d87-a2f0-d8d0d70f2137" (UID: "9ef0e388-5c57-4d87-a2f0-d8d0d70f2137"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:23:55.378907 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.378882    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-kube-api-access-bcdbq" (OuterVolumeSpecName: "kube-api-access-bcdbq") pod "9ef0e388-5c57-4d87-a2f0-d8d0d70f2137" (UID: "9ef0e388-5c57-4d87-a2f0-d8d0d70f2137"). InnerVolumeSpecName "kube-api-access-bcdbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:23:55.379020 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.378960    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ef0e388-5c57-4d87-a2f0-d8d0d70f2137" (UID: "9ef0e388-5c57-4d87-a2f0-d8d0d70f2137"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:23:55.379020 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.378981    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ef0e388-5c57-4d87-a2f0-d8d0d70f2137" (UID: "9ef0e388-5c57-4d87-a2f0-d8d0d70f2137"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:23:55.477526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.477491    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcdbq\" (UniqueName: \"kubernetes.io/projected/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-kube-api-access-bcdbq\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:23:55.477526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.477520    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:23:55.477526 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.477530    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:23:55.477804 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.477540    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:23:55.626727 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.626691    2573 generic.go:356] "Generic (PLEG): container finished" podID="9ef0e388-5c57-4d87-a2f0-d8d0d70f2137" containerID="60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509" exitCode=0
Sep 16 05:23:55.626895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.626742    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"
Sep 16 05:23:55.626895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.626756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47" event={"ID":"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137","Type":"ContainerDied","Data":"60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509"}
Sep 16 05:23:55.626895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.626804    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47" event={"ID":"9ef0e388-5c57-4d87-a2f0-d8d0d70f2137","Type":"ContainerDied","Data":"92495cc718b760f9817d6b91f8536843ff8dfdb2d30838ce7e85a29a15e70a91"}
Sep 16 05:23:55.626895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.626832    2573 scope.go:117] "RemoveContainer" containerID="60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509"
Sep 16 05:23:55.636162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.636117    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:55.636379 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.636362    2573 scope.go:117] "RemoveContainer" containerID="60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509"
Sep 16 05:23:55.636696 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:23:55.636673    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509\": container with ID starting with 60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509 not found: ID does not exist" containerID="60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509"
Sep 16 05:23:55.636802 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.636705    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509"} err="failed to get container status \"60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509\": rpc error: code = NotFound desc = could not find container \"60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509\": container with ID starting with 60ef390838fcd7082120f6bd92a24dae9b69c7d19d0481157a26873886b06509 not found: ID does not exist"
Sep 16 05:23:55.655567 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.651488    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"]
Sep 16 05:23:55.657238 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.657209    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3784e2-6768486d9c-knz47"]
Sep 16 05:23:55.740174 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:55.740136    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:56.444653 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:56.444583    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:23:56.786716 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:56.786627    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef0e388-5c57-4d87-a2f0-d8d0d70f2137" path="/var/lib/kubelet/pods/9ef0e388-5c57-4d87-a2f0-d8d0d70f2137/volumes"
Sep 16 05:23:56.866380 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:56.866355    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:23:56.870096 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:56.870072    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:23:57.890296 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:23:57.890260    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:24:01.593855 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:01.593820    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:01.694191 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:01.694151    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:01.851668 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:01.851570    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:02.933939 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:02.933902    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:03.725848 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:03.725809    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:12.634236 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:12.634197    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:12.656568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:12.656523    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"]
Sep 16 05:24:12.656837 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:12.656809    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62" podUID="31f64c4f-39a1-4090-a2a3-c7bbb896b2e0" containerName="payload-processing" containerID="cri-o://89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3" gracePeriod=30
Sep 16 05:24:12.660533 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:12.660502    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"]
Sep 16 05:24:12.661904 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:12.661849    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7" podUID="758db869-4c2f-4443-a8ff-89a913aa741d" containerName="payload-pre-processing" containerID="cri-o://f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c" gracePeriod=30
Sep 16 05:24:12.725391 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:12.725359    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:12.809364 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:12.809325    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:12.883903 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:12.883826    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:12.947028 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:12.947004    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:24:12.949539 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:12.949519    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:24:13.033588 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.033546    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-plugins-config-volume\") pod \"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0\" (UID: \"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0\") "
Sep 16 05:24:13.033815 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.033623    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758db869-4c2f-4443-a8ff-89a913aa741d-plugins-config-volume\") pod \"758db869-4c2f-4443-a8ff-89a913aa741d\" (UID: \"758db869-4c2f-4443-a8ff-89a913aa741d\") "
Sep 16 05:24:13.033815 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.033652    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wdgf\" (UniqueName: \"kubernetes.io/projected/758db869-4c2f-4443-a8ff-89a913aa741d-kube-api-access-4wdgf\") pod \"758db869-4c2f-4443-a8ff-89a913aa741d\" (UID: \"758db869-4c2f-4443-a8ff-89a913aa741d\") "
Sep 16 05:24:13.033815 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.033692    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z67nw\" (UniqueName: \"kubernetes.io/projected/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-kube-api-access-z67nw\") pod \"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0\" (UID: \"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0\") "
Sep 16 05:24:13.034069 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.034037    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31f64c4f-39a1-4090-a2a3-c7bbb896b2e0" (UID: "31f64c4f-39a1-4090-a2a3-c7bbb896b2e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:24:13.034134 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.034092    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758db869-4c2f-4443-a8ff-89a913aa741d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "758db869-4c2f-4443-a8ff-89a913aa741d" (UID: "758db869-4c2f-4443-a8ff-89a913aa741d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:24:13.035945 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.035917    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758db869-4c2f-4443-a8ff-89a913aa741d-kube-api-access-4wdgf" (OuterVolumeSpecName: "kube-api-access-4wdgf") pod "758db869-4c2f-4443-a8ff-89a913aa741d" (UID: "758db869-4c2f-4443-a8ff-89a913aa741d"). InnerVolumeSpecName "kube-api-access-4wdgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:24:13.036088 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.036071    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-kube-api-access-z67nw" (OuterVolumeSpecName: "kube-api-access-z67nw") pod "31f64c4f-39a1-4090-a2a3-c7bbb896b2e0" (UID: "31f64c4f-39a1-4090-a2a3-c7bbb896b2e0"). InnerVolumeSpecName "kube-api-access-z67nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:24:13.135162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.135115    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:24:13.135162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.135151    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758db869-4c2f-4443-a8ff-89a913aa741d-plugins-config-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:24:13.135162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.135160    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wdgf\" (UniqueName: \"kubernetes.io/projected/758db869-4c2f-4443-a8ff-89a913aa741d-kube-api-access-4wdgf\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:24:13.135162 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.135169    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z67nw\" (UniqueName: \"kubernetes.io/projected/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0-kube-api-access-z67nw\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:24:13.221880 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.221791    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:13.323026 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.322990    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:13.712467 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.712430    2573 generic.go:356] "Generic (PLEG): container finished" podID="758db869-4c2f-4443-a8ff-89a913aa741d" containerID="f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c" exitCode=0
Sep 16 05:24:13.712983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.712472    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"
Sep 16 05:24:13.712983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.712523    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7" event={"ID":"758db869-4c2f-4443-a8ff-89a913aa741d","Type":"ContainerDied","Data":"f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c"}
Sep 16 05:24:13.712983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.712586    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7" event={"ID":"758db869-4c2f-4443-a8ff-89a913aa741d","Type":"ContainerDied","Data":"d1a832572f4b76bd4c29a86995eade083df691108cf2e46c2d826b82d56f422d"}
Sep 16 05:24:13.712983 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.712639    2573 scope.go:117] "RemoveContainer" containerID="f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c"
Sep 16 05:24:13.714160 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.714062    2573 generic.go:356] "Generic (PLEG): container finished" podID="31f64c4f-39a1-4090-a2a3-c7bbb896b2e0" containerID="89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3" exitCode=0
Sep 16 05:24:13.714160 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.714116    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"
Sep 16 05:24:13.714160 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.714146    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62" event={"ID":"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0","Type":"ContainerDied","Data":"89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3"}
Sep 16 05:24:13.714332 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.714200    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62" event={"ID":"31f64c4f-39a1-4090-a2a3-c7bbb896b2e0","Type":"ContainerDied","Data":"6fc9931197373a9ce6c27981b8ee0b6c549c6f5eac79c3709c9294fc8578a81a"}
Sep 16 05:24:13.722968 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.722074    2573 scope.go:117] "RemoveContainer" containerID="f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c"
Sep 16 05:24:13.722968 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:24:13.722456    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c\": container with ID starting with f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c not found: ID does not exist" containerID="f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c"
Sep 16 05:24:13.722968 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.722491    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c"} err="failed to get container status \"f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c\": rpc error: code = NotFound desc = could not find container \"f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c\": container with ID starting with f20fb9a17217ae118d056775a897ef8d9279e08f13f882b93d3005703d00ee5c not found: ID does not exist"
Sep 16 05:24:13.722968 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.722515    2573 scope.go:117] "RemoveContainer" containerID="89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3"
Sep 16 05:24:13.731721 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.731693    2573 scope.go:117] "RemoveContainer" containerID="89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3"
Sep 16 05:24:13.732078 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:24:13.732057    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3\": container with ID starting with 89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3 not found: ID does not exist" containerID="89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3"
Sep 16 05:24:13.732134 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.732087    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3"} err="failed to get container status \"89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3\": rpc error: code = NotFound desc = could not find container \"89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3\": container with ID starting with 89ee2f9312c16418d4935ef0214126cf476e3690ed0dde8d62c64df2386d2bf3 not found: ID does not exist"
Sep 16 05:24:13.735828 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.735803    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"]
Sep 16 05:24:13.743186 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.743162    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-10ee79-78788bbfd-2csq7"]
Sep 16 05:24:13.754245 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.754219    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"]
Sep 16 05:24:13.758773 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:13.758751    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-10ee79-5cfb95c5f9-lth62"]
Sep 16 05:24:14.785923 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:14.785887    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31f64c4f-39a1-4090-a2a3-c7bbb896b2e0" path="/var/lib/kubelet/pods/31f64c4f-39a1-4090-a2a3-c7bbb896b2e0/volumes"
Sep 16 05:24:14.786298 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:14.786263    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="758db869-4c2f-4443-a8ff-89a913aa741d" path="/var/lib/kubelet/pods/758db869-4c2f-4443-a8ff-89a913aa741d/volumes"
Sep 16 05:24:15.151617 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:15.150105    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:24:19.052005 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:19.051975    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:19.114955 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:19.114925    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:19.314217 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:19.314143    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:24:23.274865 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:23.274827    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:24:46.235351 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:24:46.235304    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:25:22.569882 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:25:22.569848    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:25:23.731398 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:25:23.731359    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:25:43.445473 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:25:43.445441    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:25:43.778493 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:25:43.778405    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:25:45.123725 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:25:45.123688    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:26:09.681986 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:26:09.681949    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:26:18.193492 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:26:18.193451    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:26:24.276790 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:26:24.276710    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:26:32.680679 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:26:32.680642    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:26:51.180999 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:26:51.180962    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:26:52.759210 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:26:52.759175    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:26:52.858955 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:26:52.858919    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:26:56.976267 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:26:56.976233    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:26:58.778401 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:26:58.778362    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"]
Sep 16 05:26:58.778902 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:26:58.778584    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg" podUID="6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" containerName="manager" containerID="cri-o://08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735" gracePeriod=10
Sep 16 05:27:00.715092 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:00.715065    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 05:27:00.793403 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:00.793326    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cntz\" (UniqueName: \"kubernetes.io/projected/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-kube-api-access-6cntz\") pod \"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003\" (UID: \"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003\") "
Sep 16 05:27:00.793548 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:00.793423    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-extensions-socket-volume\") pod \"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003\" (UID: \"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003\") "
Sep 16 05:27:00.793817 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:00.793795    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" (UID: "6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:27:00.795469 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:00.795444    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-kube-api-access-6cntz" (OuterVolumeSpecName: "kube-api-access-6cntz") pod "6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" (UID: "6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003"). InnerVolumeSpecName "kube-api-access-6cntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:27:00.894200 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:00.894162    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-extensions-socket-volume\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:27:00.894200 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:00.894197    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cntz\" (UniqueName: \"kubernetes.io/projected/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003-kube-api-access-6cntz\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:27:01.484360 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.484326    2573 generic.go:356] "Generic (PLEG): container finished" podID="6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" containerID="08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735" exitCode=0
Sep 16 05:27:01.484562 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.484373    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"
Sep 16 05:27:01.484562 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.484416    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg" event={"ID":"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003","Type":"ContainerDied","Data":"08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735"}
Sep 16 05:27:01.484562 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.484476    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg" event={"ID":"6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003","Type":"ContainerDied","Data":"c8e5c6f2363f87b135f2b16c824d5dc2a9bae1393498528cf8145ad094d032a7"}
Sep 16 05:27:01.484562 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.484507    2573 scope.go:117] "RemoveContainer" containerID="08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735"
Sep 16 05:27:01.493359 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.493345    2573 scope.go:117] "RemoveContainer" containerID="8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f"
Sep 16 05:27:01.500945 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.500925    2573 scope.go:117] "RemoveContainer" containerID="08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735"
Sep 16 05:27:01.501198 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:27:01.501180    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735\": container with ID starting with 08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735 not found: ID does not exist" containerID="08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735"
Sep 16 05:27:01.501262 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.501208    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735"} err="failed to get container status \"08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735\": rpc error: code = NotFound desc = could not find container \"08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735\": container with ID starting with 08393332f1de29aa41a80b80c418bc201e11a79ca2e955e9789c8201176b8735 not found: ID does not exist"
Sep 16 05:27:01.501262 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.501225    2573 scope.go:117] "RemoveContainer" containerID="8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f"
Sep 16 05:27:01.501482 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:27:01.501467    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f\": container with ID starting with 8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f not found: ID does not exist" containerID="8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f"
Sep 16 05:27:01.501524 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.501486    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f"} err="failed to get container status \"8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f\": rpc error: code = NotFound desc = could not find container \"8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f\": container with ID starting with 8003a566c81e27a00fb45b6f05426f09cef9d5f2a644d1146835716238b7b55f not found: ID does not exist"
Sep 16 05:27:01.505532 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.505511    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"]
Sep 16 05:27:01.511159 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:01.511139    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fvkwg"]
Sep 16 05:27:02.785924 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:02.785889    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" path="/var/lib/kubelet/pods/6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003/volumes"
Sep 16 05:27:15.703895 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:15.703862    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:27:59.453408 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453375    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"]
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453678    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" containerName="manager"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453691    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" containerName="manager"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453700    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5558200-e08c-48e3-b8b1-815ce8dbd94a" containerName="maas-api"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453706    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5558200-e08c-48e3-b8b1-815ce8dbd94a" containerName="maas-api"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453713    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82bb660d-35de-472a-a09d-2e651d5d8cfb" containerName="main"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453720    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="82bb660d-35de-472a-a09d-2e651d5d8cfb" containerName="main"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453729    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e55d26-43d1-465b-a6bb-daf7aee94a58" containerName="payload-pre-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453734    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e55d26-43d1-465b-a6bb-daf7aee94a58" containerName="payload-pre-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453744    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db3fde5-25b6-4c16-bf77-3ac29075c8ee" containerName="payload-pre-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453749    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db3fde5-25b6-4c16-bf77-3ac29075c8ee" containerName="payload-pre-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453756    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316f93c1-abea-499e-851f-029a7cd16852" containerName="payload-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453761    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="316f93c1-abea-499e-851f-029a7cd16852" containerName="payload-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453771    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f64c4f-39a1-4090-a2a3-c7bbb896b2e0" containerName="payload-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453777    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f64c4f-39a1-4090-a2a3-c7bbb896b2e0" containerName="payload-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453783    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a6b5d3-da1e-4756-9d3d-3bdbccf2c426" containerName="payload-pre-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453788    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a6b5d3-da1e-4756-9d3d-3bdbccf2c426" containerName="payload-pre-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453800    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef0e388-5c57-4d87-a2f0-d8d0d70f2137" containerName="maas-api"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453805    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef0e388-5c57-4d87-a2f0-d8d0d70f2137" containerName="maas-api"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453812    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="758db869-4c2f-4443-a8ff-89a913aa741d" containerName="payload-pre-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453817    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="758db869-4c2f-4443-a8ff-89a913aa741d" containerName="payload-pre-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453824    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9376b86-7fcd-4e03-b819-6cb7490759ae" containerName="payload-processing"
Sep 16 05:27:59.453824 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453829    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9376b86-7fcd-4e03-b819-6cb7490759ae" containerName="payload-processing"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453872    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07a6b5d3-da1e-4756-9d3d-3bdbccf2c426" containerName="payload-pre-processing"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453879    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="316f93c1-abea-499e-851f-029a7cd16852" containerName="payload-processing"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453885    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e55d26-43d1-465b-a6bb-daf7aee94a58" containerName="payload-pre-processing"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453891    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="758db869-4c2f-4443-a8ff-89a913aa741d" containerName="payload-pre-processing"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453895    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="82bb660d-35de-472a-a09d-2e651d5d8cfb" containerName="main"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453901    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" containerName="manager"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453906    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" containerName="manager"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453912    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef0e388-5c57-4d87-a2f0-d8d0d70f2137" containerName="maas-api"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453918    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5558200-e08c-48e3-b8b1-815ce8dbd94a" containerName="maas-api"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453923    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db3fde5-25b6-4c16-bf77-3ac29075c8ee" containerName="payload-pre-processing"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453928    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9376b86-7fcd-4e03-b819-6cb7490759ae" containerName="payload-processing"
Sep 16 05:27:59.454568 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.453933    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f64c4f-39a1-4090-a2a3-c7bbb896b2e0" containerName="payload-processing"
Sep 16 05:27:59.456872 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.456854    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"
Sep 16 05:27:59.458986 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.458962    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l67nf\""
Sep 16 05:27:59.467681 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.467653    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"]
Sep 16 05:27:59.566871 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.566846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8zk\" (UniqueName: \"kubernetes.io/projected/29217f01-a1a4-4061-8002-2ea00caca8cd-kube-api-access-6c8zk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24nhz\" (UID: \"29217f01-a1a4-4061-8002-2ea00caca8cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"
Sep 16 05:27:59.567004 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.566895    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29217f01-a1a4-4061-8002-2ea00caca8cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24nhz\" (UID: \"29217f01-a1a4-4061-8002-2ea00caca8cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"
Sep 16 05:27:59.668204 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.668175    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29217f01-a1a4-4061-8002-2ea00caca8cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24nhz\" (UID: \"29217f01-a1a4-4061-8002-2ea00caca8cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"
Sep 16 05:27:59.668335 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.668220    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8zk\" (UniqueName: \"kubernetes.io/projected/29217f01-a1a4-4061-8002-2ea00caca8cd-kube-api-access-6c8zk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24nhz\" (UID: \"29217f01-a1a4-4061-8002-2ea00caca8cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"
Sep 16 05:27:59.668566 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.668551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29217f01-a1a4-4061-8002-2ea00caca8cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24nhz\" (UID: \"29217f01-a1a4-4061-8002-2ea00caca8cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"
Sep 16 05:27:59.676230 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.676203    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8zk\" (UniqueName: \"kubernetes.io/projected/29217f01-a1a4-4061-8002-2ea00caca8cd-kube-api-access-6c8zk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24nhz\" (UID: \"29217f01-a1a4-4061-8002-2ea00caca8cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"
Sep 16 05:27:59.767446 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:27:59.767354    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"
Sep 16 05:28:00.046109 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:00.046044    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"]
Sep 16 05:28:00.049030 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:28:00.049004    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29217f01_a1a4_4061_8002_2ea00caca8cd.slice/crio-15f4ea22286d7a6de4c5ef1fac9dfd4481e089b0d6e2c4993bd0631a04da18fb WatchSource:0}: Error finding container 15f4ea22286d7a6de4c5ef1fac9dfd4481e089b0d6e2c4993bd0631a04da18fb: Status 404 returned error can't find the container with id 15f4ea22286d7a6de4c5ef1fac9dfd4481e089b0d6e2c4993bd0631a04da18fb
Sep 16 05:28:00.742631 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:00.742577    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz" event={"ID":"29217f01-a1a4-4061-8002-2ea00caca8cd","Type":"ContainerStarted","Data":"24cd79f60e7f982886fcf6b7863ec0943bdf0a10a34cc33f38c880e8914b5db8"}
Sep 16 05:28:00.743013 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:00.742657    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"
Sep 16 05:28:00.743013 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:00.742675    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz" event={"ID":"29217f01-a1a4-4061-8002-2ea00caca8cd","Type":"ContainerStarted","Data":"15f4ea22286d7a6de4c5ef1fac9dfd4481e089b0d6e2c4993bd0631a04da18fb"}
Sep 16 05:28:00.766035 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:00.765981    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz" podStartSLOduration=1.76596618 podStartE2EDuration="1.76596618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:28:00.763827144 +0000 UTC m=+2344.429756780" watchObservedRunningTime="2026-09-16 05:28:00.76596618 +0000 UTC m=+2344.431895773"
Sep 16 05:28:11.749916 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:11.749877    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24nhz"
Sep 16 05:28:23.476257 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:23.476216    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:28:30.775690 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:30.775654    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:28:36.571402 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:36.571368    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:28:40.642320 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:40.642270    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:28:43.247400 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.247364    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"]
Sep 16 05:28:43.247878 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.247700    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" containerName="manager"
Sep 16 05:28:43.247878 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.247715    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6b0b0f-ecfc-4441-8ac3-f37f11cf3003" containerName="manager"
Sep 16 05:28:43.250817 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.250797    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.252969 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.252942    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bd9fd524-kserve-self-signed-certs\""
Sep 16 05:28:43.258708 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.258683    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"]
Sep 16 05:28:43.270069 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.270043    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:28:43.350557 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.350516    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-dshm\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.350557 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.350559    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tmp-dir\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.350788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.350588    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-home\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.350788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.350656    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tls-certs\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.350788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.350705    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-model-cache\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.350788 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.350736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jn8\" (UniqueName: \"kubernetes.io/projected/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-kube-api-access-s5jn8\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.451117 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.451062    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-model-cache\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.451117 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.451117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jn8\" (UniqueName: \"kubernetes.io/projected/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-kube-api-access-s5jn8\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.451331 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.451146    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-dshm\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.451331 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.451164    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tmp-dir\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.451331 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.451191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-home\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.451468 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.451345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tls-certs\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.451528 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.451507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-model-cache\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.451665 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.451644    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tmp-dir\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.451757 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.451694    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-home\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.453492 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.453470    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-dshm\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.453976 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.453942    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tls-certs\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.458910 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.458888    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jn8\" (UniqueName: \"kubernetes.io/projected/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-kube-api-access-s5jn8\") pod \"e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") " pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.561125 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.561048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:43.833584 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.833555    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"]
Sep 16 05:28:43.836285 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:28:43.836247    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod928aa099_aa4b_4a7f_8c42_fc4adadd69f7.slice/crio-7183bb08a4240a619553f788bcbecde01843ebcc9564245888994cb8e4c2e905 WatchSource:0}: Error finding container 7183bb08a4240a619553f788bcbecde01843ebcc9564245888994cb8e4c2e905: Status 404 returned error can't find the container with id 7183bb08a4240a619553f788bcbecde01843ebcc9564245888994cb8e4c2e905
Sep 16 05:28:43.935327 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.935291    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn" event={"ID":"928aa099-aa4b-4a7f-8c42-fc4adadd69f7","Type":"ContainerStarted","Data":"b0277c262d6716c94822f6cc02b506e8704d5483fabdb7e2e91fcb94c0298705"}
Sep 16 05:28:43.935327 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.935330    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn" event={"ID":"928aa099-aa4b-4a7f-8c42-fc4adadd69f7","Type":"ContainerStarted","Data":"7183bb08a4240a619553f788bcbecde01843ebcc9564245888994cb8e4c2e905"}
Sep 16 05:28:43.950814 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:43.950738    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn" podStartSLOduration=0.950715593 podStartE2EDuration="950.715593ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:28:43.949820503 +0000 UTC m=+2387.615750100" watchObservedRunningTime="2026-09-16 05:28:43.950715593 +0000 UTC m=+2387.616645190"
Sep 16 05:28:45.790337 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:45.790303    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:28:53.562104 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:53.562068    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:53.562588 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:53.562146    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:53.574634 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:53.574584    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:53.993459 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:53.993431    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:28:56.890629 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:56.890582    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:28:56.902292 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:28:56.902265    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w4q88_6985599b-f750-48f3-9e34-55dd74ed94bc/ovn-acl-logging/0.log"
Sep 16 05:29:00.330731 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.330693    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"]
Sep 16 05:29:00.334279 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.334255    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.336430 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.336408    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bd9fd524-kserve-self-signed-certs\""
Sep 16 05:29:00.342976 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.342954    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"]
Sep 16 05:29:00.368208 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.368178    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:29:00.398665 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.398631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-tmp-dir\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.398665 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.398667    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc4ba15-1b53-4776-8399-7d36b40c7445-tls-certs\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.398893 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.398698    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-model-cache\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.398893 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.398746    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-dshm\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.398893 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.398775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-home\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.398893 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.398797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54lkf\" (UniqueName: \"kubernetes.io/projected/cdc4ba15-1b53-4776-8399-7d36b40c7445-kube-api-access-54lkf\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.500109 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.500056    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-dshm\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.500109 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.500124    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-home\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.500393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.500142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54lkf\" (UniqueName: \"kubernetes.io/projected/cdc4ba15-1b53-4776-8399-7d36b40c7445-kube-api-access-54lkf\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.500393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.500183    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-tmp-dir\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.500393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.500200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc4ba15-1b53-4776-8399-7d36b40c7445-tls-certs\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.500393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.500359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-model-cache\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.500640 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.500622    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-tmp-dir\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.500689 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.500649    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-home\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.500728 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.500718    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-model-cache\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.502504 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.502482    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-dshm\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.502785 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.502769    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc4ba15-1b53-4776-8399-7d36b40c7445-tls-certs\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.507219 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.507192    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54lkf\" (UniqueName: \"kubernetes.io/projected/cdc4ba15-1b53-4776-8399-7d36b40c7445-kube-api-access-54lkf\") pod \"e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") " pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.645044 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.645012    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:00.920008 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:00.919926    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"]
Sep 16 05:29:00.922863 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:29:00.922831    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdc4ba15_1b53_4776_8399_7d36b40c7445.slice/crio-1396875a194df04e8056b9247c3dd6ea4ac7a19ca8602ace6b943e1811052e83 WatchSource:0}: Error finding container 1396875a194df04e8056b9247c3dd6ea4ac7a19ca8602ace6b943e1811052e83: Status 404 returned error can't find the container with id 1396875a194df04e8056b9247c3dd6ea4ac7a19ca8602ace6b943e1811052e83
Sep 16 05:29:01.022234 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:01.022196    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf" event={"ID":"cdc4ba15-1b53-4776-8399-7d36b40c7445","Type":"ContainerStarted","Data":"db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce"}
Sep 16 05:29:01.022385 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:01.022244    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf" event={"ID":"cdc4ba15-1b53-4776-8399-7d36b40c7445","Type":"ContainerStarted","Data":"1396875a194df04e8056b9247c3dd6ea4ac7a19ca8602ace6b943e1811052e83"}
Sep 16 05:29:01.038740 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:01.038688    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf" podStartSLOduration=1.038671482 podStartE2EDuration="1.038671482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:29:01.03785796 +0000 UTC m=+2404.703787578" watchObservedRunningTime="2026-09-16 05:29:01.038671482 +0000 UTC m=+2404.704601074"
Sep 16 05:29:03.281358 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:03.281317    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:29:10.645670 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:10.645627    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:10.646172 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:10.645772    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:10.658197 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:10.658174    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:11.078993 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:11.078912    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:24.001115 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:24.001034    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"]
Sep 16 05:29:24.021076 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:24.021035    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:29:25.564128 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:25.564092    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"]
Sep 16 05:29:25.584378 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:25.584345    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:29:26.080216 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:26.080180    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:29:35.182628 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:35.182581    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:29:39.020760 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.020690    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn" podUID="928aa099-aa4b-4a7f-8c42-fc4adadd69f7" containerName="main" containerID="cri-o://b0277c262d6716c94822f6cc02b506e8704d5483fabdb7e2e91fcb94c0298705" gracePeriod=45
Sep 16 05:29:39.196263 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.196232    2573 generic.go:356] "Generic (PLEG): container finished" podID="928aa099-aa4b-4a7f-8c42-fc4adadd69f7" containerID="b0277c262d6716c94822f6cc02b506e8704d5483fabdb7e2e91fcb94c0298705" exitCode=0
Sep 16 05:29:39.196415 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.196302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn" event={"ID":"928aa099-aa4b-4a7f-8c42-fc4adadd69f7","Type":"ContainerDied","Data":"b0277c262d6716c94822f6cc02b506e8704d5483fabdb7e2e91fcb94c0298705"}
Sep 16 05:29:39.270744 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.270719    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:29:39.347155 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.347061    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tls-certs\") pod \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") "
Sep 16 05:29:39.347155 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.347119    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5jn8\" (UniqueName: \"kubernetes.io/projected/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-kube-api-access-s5jn8\") pod \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") "
Sep 16 05:29:39.347155 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.347148    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tmp-dir\") pod \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") "
Sep 16 05:29:39.347438 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.347185    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-dshm\") pod \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") "
Sep 16 05:29:39.347438 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.347211    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-model-cache\") pod \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") "
Sep 16 05:29:39.347438 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.347260    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-home\") pod \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\" (UID: \"928aa099-aa4b-4a7f-8c42-fc4adadd69f7\") "
Sep 16 05:29:39.347622 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.347504    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "928aa099-aa4b-4a7f-8c42-fc4adadd69f7" (UID: "928aa099-aa4b-4a7f-8c42-fc4adadd69f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:29:39.347705 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.347591    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "928aa099-aa4b-4a7f-8c42-fc4adadd69f7" (UID: "928aa099-aa4b-4a7f-8c42-fc4adadd69f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:29:39.347705 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.347642    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-home" (OuterVolumeSpecName: "home") pod "928aa099-aa4b-4a7f-8c42-fc4adadd69f7" (UID: "928aa099-aa4b-4a7f-8c42-fc4adadd69f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:29:39.349530 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.349509    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "928aa099-aa4b-4a7f-8c42-fc4adadd69f7" (UID: "928aa099-aa4b-4a7f-8c42-fc4adadd69f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:29:39.349919 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.349896    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-dshm" (OuterVolumeSpecName: "dshm") pod "928aa099-aa4b-4a7f-8c42-fc4adadd69f7" (UID: "928aa099-aa4b-4a7f-8c42-fc4adadd69f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:29:39.349971 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.349912    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-kube-api-access-s5jn8" (OuterVolumeSpecName: "kube-api-access-s5jn8") pod "928aa099-aa4b-4a7f-8c42-fc4adadd69f7" (UID: "928aa099-aa4b-4a7f-8c42-fc4adadd69f7"). InnerVolumeSpecName "kube-api-access-s5jn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:29:39.448352 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.448311    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:39.448352 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.448346    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jn8\" (UniqueName: \"kubernetes.io/projected/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-kube-api-access-s5jn8\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:39.448352 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.448356    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:39.448352 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.448365    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:39.448652 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.448376    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:39.448652 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:39.448387    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928aa099-aa4b-4a7f-8c42-fc4adadd69f7-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:40.201925 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.201892    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"
Sep 16 05:29:40.204334 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.201930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn" event={"ID":"928aa099-aa4b-4a7f-8c42-fc4adadd69f7","Type":"ContainerDied","Data":"7183bb08a4240a619553f788bcbecde01843ebcc9564245888994cb8e4c2e905"}
Sep 16 05:29:40.204334 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.201981    2573 scope.go:117] "RemoveContainer" containerID="b0277c262d6716c94822f6cc02b506e8704d5483fabdb7e2e91fcb94c0298705"
Sep 16 05:29:40.223097 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.223070    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"]
Sep 16 05:29:40.227232 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.227209    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bd9fd524-kserve-c95d46d99-kt5hn"]
Sep 16 05:29:40.584761 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.584640    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf" podUID="cdc4ba15-1b53-4776-8399-7d36b40c7445" containerName="main" containerID="cri-o://db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce" gracePeriod=45
Sep 16 05:29:40.786739 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.786705    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="928aa099-aa4b-4a7f-8c42-fc4adadd69f7" path="/var/lib/kubelet/pods/928aa099-aa4b-4a7f-8c42-fc4adadd69f7/volumes"
Sep 16 05:29:40.836436 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.836378    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:40.860297 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.860261    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54lkf\" (UniqueName: \"kubernetes.io/projected/cdc4ba15-1b53-4776-8399-7d36b40c7445-kube-api-access-54lkf\") pod \"cdc4ba15-1b53-4776-8399-7d36b40c7445\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") "
Sep 16 05:29:40.860478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.860304    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc4ba15-1b53-4776-8399-7d36b40c7445-tls-certs\") pod \"cdc4ba15-1b53-4776-8399-7d36b40c7445\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") "
Sep 16 05:29:40.860478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.860338    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-home\") pod \"cdc4ba15-1b53-4776-8399-7d36b40c7445\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") "
Sep 16 05:29:40.860478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.860361    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-tmp-dir\") pod \"cdc4ba15-1b53-4776-8399-7d36b40c7445\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") "
Sep 16 05:29:40.860478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.860429    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-dshm\") pod \"cdc4ba15-1b53-4776-8399-7d36b40c7445\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") "
Sep 16 05:29:40.860478 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.860459    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-model-cache\") pod \"cdc4ba15-1b53-4776-8399-7d36b40c7445\" (UID: \"cdc4ba15-1b53-4776-8399-7d36b40c7445\") "
Sep 16 05:29:40.860793 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.860671    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-home" (OuterVolumeSpecName: "home") pod "cdc4ba15-1b53-4776-8399-7d36b40c7445" (UID: "cdc4ba15-1b53-4776-8399-7d36b40c7445"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:29:40.860793 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.860754    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-home\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:40.860916 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.860892    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdc4ba15-1b53-4776-8399-7d36b40c7445" (UID: "cdc4ba15-1b53-4776-8399-7d36b40c7445"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:29:40.860986 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.860958    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdc4ba15-1b53-4776-8399-7d36b40c7445" (UID: "cdc4ba15-1b53-4776-8399-7d36b40c7445"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:29:40.862579 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.862541    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-dshm" (OuterVolumeSpecName: "dshm") pod "cdc4ba15-1b53-4776-8399-7d36b40c7445" (UID: "cdc4ba15-1b53-4776-8399-7d36b40c7445"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 05:29:40.862579 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.862570    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc4ba15-1b53-4776-8399-7d36b40c7445-kube-api-access-54lkf" (OuterVolumeSpecName: "kube-api-access-54lkf") pod "cdc4ba15-1b53-4776-8399-7d36b40c7445" (UID: "cdc4ba15-1b53-4776-8399-7d36b40c7445"). InnerVolumeSpecName "kube-api-access-54lkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:29:40.862791 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.862740    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc4ba15-1b53-4776-8399-7d36b40c7445-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdc4ba15-1b53-4776-8399-7d36b40c7445" (UID: "cdc4ba15-1b53-4776-8399-7d36b40c7445"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:29:40.962017 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.961977    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc4ba15-1b53-4776-8399-7d36b40c7445-tls-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:40.962017 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.962007    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-tmp-dir\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:40.962017 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.962016    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-dshm\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:40.962017 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.962024    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc4ba15-1b53-4776-8399-7d36b40c7445-model-cache\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:40.962282 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:40.962033    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54lkf\" (UniqueName: \"kubernetes.io/projected/cdc4ba15-1b53-4776-8399-7d36b40c7445-kube-api-access-54lkf\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:41.213158 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:41.213117    2573 generic.go:356] "Generic (PLEG): container finished" podID="cdc4ba15-1b53-4776-8399-7d36b40c7445" containerID="db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce" exitCode=0
Sep 16 05:29:41.213672 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:41.213176    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"
Sep 16 05:29:41.213672 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:41.213197    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf" event={"ID":"cdc4ba15-1b53-4776-8399-7d36b40c7445","Type":"ContainerDied","Data":"db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce"}
Sep 16 05:29:41.213672 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:41.213239    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf" event={"ID":"cdc4ba15-1b53-4776-8399-7d36b40c7445","Type":"ContainerDied","Data":"1396875a194df04e8056b9247c3dd6ea4ac7a19ca8602ace6b943e1811052e83"}
Sep 16 05:29:41.213672 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:41.213259    2573 scope.go:117] "RemoveContainer" containerID="db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce"
Sep 16 05:29:41.223321 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:41.223304    2573 scope.go:117] "RemoveContainer" containerID="db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce"
Sep 16 05:29:41.223617 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:29:41.223584    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce\": container with ID starting with db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce not found: ID does not exist" containerID="db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce"
Sep 16 05:29:41.223705 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:41.223640    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce"} err="failed to get container status \"db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce\": rpc error: code = NotFound desc = could not find container \"db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce\": container with ID starting with db09d7c48605c8f447a161de6dca73067a43a05866b38f837852393b03962dce not found: ID does not exist"
Sep 16 05:29:41.233754 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:41.233730    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"]
Sep 16 05:29:41.235709 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:41.235686    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bd9fd524-kserve-657dff89cf-bsxmf"]
Sep 16 05:29:42.787959 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:42.787924    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdc4ba15-1b53-4776-8399-7d36b40c7445" path="/var/lib/kubelet/pods/cdc4ba15-1b53-4776-8399-7d36b40c7445/volumes"
Sep 16 05:29:44.179314 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.179275    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:29:44.396446 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.396413    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"]
Sep 16 05:29:44.396742 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.396730    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc4ba15-1b53-4776-8399-7d36b40c7445" containerName="main"
Sep 16 05:29:44.396742 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.396743    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc4ba15-1b53-4776-8399-7d36b40c7445" containerName="main"
Sep 16 05:29:44.396832 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.396754    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="928aa099-aa4b-4a7f-8c42-fc4adadd69f7" containerName="main"
Sep 16 05:29:44.396832 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.396760    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="928aa099-aa4b-4a7f-8c42-fc4adadd69f7" containerName="main"
Sep 16 05:29:44.396832 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.396816    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdc4ba15-1b53-4776-8399-7d36b40c7445" containerName="main"
Sep 16 05:29:44.396832 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.396825    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="928aa099-aa4b-4a7f-8c42-fc4adadd69f7" containerName="main"
Sep 16 05:29:44.400699 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.400679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.402874 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.402848    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 05:29:44.403725 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.403701    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 05:29:44.403725 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.403720    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 05:29:44.403911 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.403703    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qdjvt\""
Sep 16 05:29:44.403911 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.403744    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 05:29:44.411293 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.411268    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"]
Sep 16 05:29:44.413736 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.413716    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"]
Sep 16 05:29:44.416932 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.416916    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.428657 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.428448    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"]
Sep 16 05:29:44.494071 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.493975    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1934ef5f-76b0-4b6e-b506-1306bd6adbca-metrics-certs\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.494071 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.494012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1934ef5f-76b0-4b6e-b506-1306bd6adbca-maas-api-tls\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.494071 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.494040    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1934ef5f-76b0-4b6e-b506-1306bd6adbca-openshift-service-ca\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.494071 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.494071    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdj2j\" (UniqueName: \"kubernetes.io/projected/1934ef5f-76b0-4b6e-b506-1306bd6adbca-kube-api-access-vdj2j\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.494391 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.494096    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7sx\" (UniqueName: \"kubernetes.io/projected/1235a4f1-29a3-4b8f-98c2-c357e6986958-kube-api-access-8x7sx\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.494391 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.494121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-metrics-certs\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.494391 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.494184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-maas-api-tls\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.494391 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.494227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235a4f1-29a3-4b8f-98c2-c357e6986958-openshift-service-ca\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.595323 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.595271    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-maas-api-tls\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.595323 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.595332    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235a4f1-29a3-4b8f-98c2-c357e6986958-openshift-service-ca\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.595579 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.595368    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1934ef5f-76b0-4b6e-b506-1306bd6adbca-metrics-certs\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.595579 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.595386    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1934ef5f-76b0-4b6e-b506-1306bd6adbca-maas-api-tls\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.595579 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.595407    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1934ef5f-76b0-4b6e-b506-1306bd6adbca-openshift-service-ca\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.595579 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.595428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdj2j\" (UniqueName: \"kubernetes.io/projected/1934ef5f-76b0-4b6e-b506-1306bd6adbca-kube-api-access-vdj2j\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.595579 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.595454    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7sx\" (UniqueName: \"kubernetes.io/projected/1235a4f1-29a3-4b8f-98c2-c357e6986958-kube-api-access-8x7sx\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.595579 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.595487    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-metrics-certs\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.596343 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.596311    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1934ef5f-76b0-4b6e-b506-1306bd6adbca-openshift-service-ca\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.596559 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.596538    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235a4f1-29a3-4b8f-98c2-c357e6986958-openshift-service-ca\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.598098 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.598075    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1934ef5f-76b0-4b6e-b506-1306bd6adbca-maas-api-tls\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.598252 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.598233    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-metrics-certs\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.598442 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.598424    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-maas-api-tls\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.598524 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.598504    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1934ef5f-76b0-4b6e-b506-1306bd6adbca-metrics-certs\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.602709 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.602690    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdj2j\" (UniqueName: \"kubernetes.io/projected/1934ef5f-76b0-4b6e-b506-1306bd6adbca-kube-api-access-vdj2j\") pod \"maas-api-5c4458d669-qhz62\" (UID: \"1934ef5f-76b0-4b6e-b506-1306bd6adbca\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.603042 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.603021    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7sx\" (UniqueName: \"kubernetes.io/projected/1235a4f1-29a3-4b8f-98c2-c357e6986958-kube-api-access-8x7sx\") pod \"maas-api-86c4f95b8f-d52fj\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.712213 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.712174    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:44.727404 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.727379    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:44.990383 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.990346    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"]
Sep 16 05:29:44.992915 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:29:44.992881    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1235a4f1_29a3_4b8f_98c2_c357e6986958.slice/crio-b513f052c1fcdd88232a82321920d7322e023390d298d5d47a26e3ad1144c26a WatchSource:0}: Error finding container b513f052c1fcdd88232a82321920d7322e023390d298d5d47a26e3ad1144c26a: Status 404 returned error can't find the container with id b513f052c1fcdd88232a82321920d7322e023390d298d5d47a26e3ad1144c26a
Sep 16 05:29:44.994587 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:44.994562    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 05:29:45.007422 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:45.007402    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"]
Sep 16 05:29:45.009632 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:29:45.009606    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1934ef5f_76b0_4b6e_b506_1306bd6adbca.slice/crio-c72a33a0c92ff321f6802744e701bf5c1b3df003730cee15b07c1cf59d0cb221 WatchSource:0}: Error finding container c72a33a0c92ff321f6802744e701bf5c1b3df003730cee15b07c1cf59d0cb221: Status 404 returned error can't find the container with id c72a33a0c92ff321f6802744e701bf5c1b3df003730cee15b07c1cf59d0cb221
Sep 16 05:29:45.239152 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:45.239112    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62" event={"ID":"1934ef5f-76b0-4b6e-b506-1306bd6adbca","Type":"ContainerStarted","Data":"c72a33a0c92ff321f6802744e701bf5c1b3df003730cee15b07c1cf59d0cb221"}
Sep 16 05:29:45.240441 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:45.240405    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj" event={"ID":"1235a4f1-29a3-4b8f-98c2-c357e6986958","Type":"ContainerStarted","Data":"b513f052c1fcdd88232a82321920d7322e023390d298d5d47a26e3ad1144c26a"}
Sep 16 05:29:46.246799 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:46.246761    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62" event={"ID":"1934ef5f-76b0-4b6e-b506-1306bd6adbca","Type":"ContainerStarted","Data":"391f7f55ed04a12d63252318039566d0dccdfe4f6c6b8c6c132198dffab3577e"}
Sep 16 05:29:46.247273 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:46.246899    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:46.248418 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:46.248394    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj" event={"ID":"1235a4f1-29a3-4b8f-98c2-c357e6986958","Type":"ContainerStarted","Data":"d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223"}
Sep 16 05:29:46.248534 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:46.248523    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:46.263868 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:46.263812    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62" podStartSLOduration=1.852680044 podStartE2EDuration="2.263773759s" totalImagesPullingTime="411.093715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:29:46.262876355 +0000 UTC m=+2449.928805947" watchObservedRunningTime="2026-09-16 05:29:46.263773759 +0000 UTC m=+2449.929703353"
Sep 16 05:29:46.279154 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:46.279102    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj" podStartSLOduration=1.79914632 podStartE2EDuration="2.279087771s" totalImagesPullingTime="479.941451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:29:46.278171578 +0000 UTC m=+2449.944101196" watchObservedRunningTime="2026-09-16 05:29:46.279087771 +0000 UTC m=+2449.945017365"
Sep 16 05:29:51.879268 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:51.879232    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:29:52.258702 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.258623    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:52.258902 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.258884    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c4458d669-qhz62"
Sep 16 05:29:52.320007 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.319976    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"]
Sep 16 05:29:52.320239 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.320197    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj" podUID="1235a4f1-29a3-4b8f-98c2-c357e6986958" containerName="maas-api" containerID="cri-o://d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223" gracePeriod=30
Sep 16 05:29:52.572945 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.572914    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:52.665934 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.665900    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-metrics-certs\") pod \"1235a4f1-29a3-4b8f-98c2-c357e6986958\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") "
Sep 16 05:29:52.666131 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.665959    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235a4f1-29a3-4b8f-98c2-c357e6986958-openshift-service-ca\") pod \"1235a4f1-29a3-4b8f-98c2-c357e6986958\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") "
Sep 16 05:29:52.666131 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.666043    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x7sx\" (UniqueName: \"kubernetes.io/projected/1235a4f1-29a3-4b8f-98c2-c357e6986958-kube-api-access-8x7sx\") pod \"1235a4f1-29a3-4b8f-98c2-c357e6986958\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") "
Sep 16 05:29:52.666131 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.666072    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-maas-api-tls\") pod \"1235a4f1-29a3-4b8f-98c2-c357e6986958\" (UID: \"1235a4f1-29a3-4b8f-98c2-c357e6986958\") "
Sep 16 05:29:52.666373 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.666345    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1235a4f1-29a3-4b8f-98c2-c357e6986958-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1235a4f1-29a3-4b8f-98c2-c357e6986958" (UID: "1235a4f1-29a3-4b8f-98c2-c357e6986958"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 05:29:52.668203 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.668174    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1235a4f1-29a3-4b8f-98c2-c357e6986958" (UID: "1235a4f1-29a3-4b8f-98c2-c357e6986958"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:29:52.668317 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.668240    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1235a4f1-29a3-4b8f-98c2-c357e6986958-kube-api-access-8x7sx" (OuterVolumeSpecName: "kube-api-access-8x7sx") pod "1235a4f1-29a3-4b8f-98c2-c357e6986958" (UID: "1235a4f1-29a3-4b8f-98c2-c357e6986958"). InnerVolumeSpecName "kube-api-access-8x7sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 05:29:52.668317 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.668307    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1235a4f1-29a3-4b8f-98c2-c357e6986958" (UID: "1235a4f1-29a3-4b8f-98c2-c357e6986958"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 05:29:52.766931 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.766893    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235a4f1-29a3-4b8f-98c2-c357e6986958-openshift-service-ca\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:52.766931 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.766924    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7sx\" (UniqueName: \"kubernetes.io/projected/1235a4f1-29a3-4b8f-98c2-c357e6986958-kube-api-access-8x7sx\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:52.766931 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.766934    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-maas-api-tls\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:52.766931 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:52.766944    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235a4f1-29a3-4b8f-98c2-c357e6986958-metrics-certs\") on node \"ip-10-0-133-117.ec2.internal\" DevicePath \"\""
Sep 16 05:29:53.283040 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:53.283007    2573 generic.go:356] "Generic (PLEG): container finished" podID="1235a4f1-29a3-4b8f-98c2-c357e6986958" containerID="d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223" exitCode=0
Sep 16 05:29:53.283457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:53.283054    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"
Sep 16 05:29:53.283457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:53.283094    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj" event={"ID":"1235a4f1-29a3-4b8f-98c2-c357e6986958","Type":"ContainerDied","Data":"d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223"}
Sep 16 05:29:53.283457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:53.283149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj" event={"ID":"1235a4f1-29a3-4b8f-98c2-c357e6986958","Type":"ContainerDied","Data":"b513f052c1fcdd88232a82321920d7322e023390d298d5d47a26e3ad1144c26a"}
Sep 16 05:29:53.283457 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:53.283171    2573 scope.go:117] "RemoveContainer" containerID="d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223"
Sep 16 05:29:53.291905 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:53.291880    2573 scope.go:117] "RemoveContainer" containerID="d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223"
Sep 16 05:29:53.292228 ip-10-0-133-117 kubenswrapper[2573]: E0916 05:29:53.292208    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223\": container with ID starting with d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223 not found: ID does not exist" containerID="d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223"
Sep 16 05:29:53.292276 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:53.292238    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223"} err="failed to get container status \"d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223\": rpc error: code = NotFound desc = could not find container \"d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223\": container with ID starting with d9ef9f1efc37df18394b63a96b0c5b138e3d803a292e3346195c7980ba9e3223 not found: ID does not exist"
Sep 16 05:29:53.297404 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:53.297377    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"]
Sep 16 05:29:53.302238 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:53.302210    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-d52fj"]
Sep 16 05:29:54.786420 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:54.786385    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1235a4f1-29a3-4b8f-98c2-c357e6986958" path="/var/lib/kubelet/pods/1235a4f1-29a3-4b8f-98c2-c357e6986958/volumes"
Sep 16 05:29:58.877153 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:29:58.877110    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:30:04.972541 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:04.972497    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:30:11.971868 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:11.971834    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-prqbz"]
Sep 16 05:30:19.771514 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:19.771482    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-c8zqs_2339d094-fc27-46a3-bd1a-8b3b738dcbd2/server/0.log"
Sep 16 05:30:19.889615 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:19.889570    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lxpdd_532d8a7e-b308-41f0-8779-351326953242/manager/0.log"
Sep 16 05:30:20.130540 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:20.130509    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756589c867-w899b_b8661495-0d52-469e-b831-f10980137a67/copy-manifests/0.log"
Sep 16 05:30:20.136837 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:20.136812    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756589c867-w899b_b8661495-0d52-469e-b831-f10980137a67/manager/0.log"
Sep 16 05:30:20.720770 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:20.720715    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c4458d669-qhz62_1934ef5f-76b0-4b6e-b506-1306bd6adbca/maas-api/0.log"
Sep 16 05:30:20.843426 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:20.843393    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825595-l9tpn_d3964d8d-8453-4d80-9dfc-d66d2127aeba/cleanup/0.log"
Sep 16 05:30:22.300021 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:22.299991    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-24nhz_29217f01-a1a4-4061-8002-2ea00caca8cd/manager/0.log"
Sep 16 05:30:22.429177 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:22.429149    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-prqbz_917904f2-ff0d-49fe-bcf4-774129f33a89/limitador/0.log"
Sep 16 05:30:22.543936 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:22.543903    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-rds7k_9224eeff-9a2a-46e5-977b-2a0cdd96d9db/manager/0.log"
Sep 16 05:30:23.022101 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:23.022072    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lm4ts_04d91007-8ea8-4099-96ca-942aafbd5768/discovery/0.log"
Sep 16 05:30:23.237393 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:23.237364    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6965d5747b-dsn8t_dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd/kube-auth-proxy/0.log"
Sep 16 05:30:24.167873 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:24.167838    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vr5dh_38175809-aafd-43ee-9db6-719d5f516f7c/main/0.log"
Sep 16 05:30:24.493642 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:24.493552    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flczgk_471b1d22-1296-4f99-aff8-a9a1c0e4604e/main/0.log"
Sep 16 05:30:29.872931 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:29.872898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:30:31.247793 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:31.247757    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log" limit=1024
Sep 16 05:30:32.842650 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:32.842620    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vqmw5_b9705e01-c8d0-481f-8769-ffd767bc202c/global-pull-secret-syncer/0.log"
Sep 16 05:30:32.887975 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:32.887943    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-24mv8_36a9d2ea-0102-4624-bfa6-b96860b1f483/konnectivity-agent/0.log"
Sep 16 05:30:32.978228 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:32.978204    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-117.ec2.internal_461b5598d7024278d7832d79de8a2c73/haproxy/0.log"
Sep 16 05:30:37.690655 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:37.690625    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-24nhz_29217f01-a1a4-4061-8002-2ea00caca8cd/manager/0.log"
Sep 16 05:30:37.717566 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:37.717537    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-prqbz_917904f2-ff0d-49fe-bcf4-774129f33a89/limitador/0.log"
Sep 16 05:30:37.771740 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:37.771669    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-rds7k_9224eeff-9a2a-46e5-977b-2a0cdd96d9db/manager/0.log"
Sep 16 05:30:39.572973 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:39.572947    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dqswl_15c4e760-06e6-4df7-8cea-2cc464744f78/node-exporter/0.log"
Sep 16 05:30:39.592335 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:39.592313    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dqswl_15c4e760-06e6-4df7-8cea-2cc464744f78/kube-rbac-proxy/0.log"
Sep 16 05:30:39.611099 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:39.611078    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dqswl_15c4e760-06e6-4df7-8cea-2cc464744f78/init-textfile/0.log"
Sep 16 05:30:43.396308 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.396265    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"]
Sep 16 05:30:43.396711 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.396562    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1235a4f1-29a3-4b8f-98c2-c357e6986958" containerName="maas-api"
Sep 16 05:30:43.396711 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.396573    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1235a4f1-29a3-4b8f-98c2-c357e6986958" containerName="maas-api"
Sep 16 05:30:43.396711 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.396651    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1235a4f1-29a3-4b8f-98c2-c357e6986958" containerName="maas-api"
Sep 16 05:30:43.399838 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.399818    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.401891 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.401864    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hf662\"/\"openshift-service-ca.crt\""
Sep 16 05:30:43.402013 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.401870    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hf662\"/\"kube-root-ca.crt\""
Sep 16 05:30:43.402548 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.402528    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hf662\"/\"default-dockercfg-p95kz\""
Sep 16 05:30:43.409028 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.409007    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"]
Sep 16 05:30:43.480716 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.480686    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-podres\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.480850 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.480721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-lib-modules\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.480850 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.480742    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-proc\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.480850 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.480759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l9pk\" (UniqueName: \"kubernetes.io/projected/33b459ea-a712-43f4-b8d6-48ee362668aa-kube-api-access-6l9pk\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.480850 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.480817    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-sys\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.581656 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.581626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-podres\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.581786 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.581661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-lib-modules\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.581786 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.581684    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-proc\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.581786 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.581710    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l9pk\" (UniqueName: \"kubernetes.io/projected/33b459ea-a712-43f4-b8d6-48ee362668aa-kube-api-access-6l9pk\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.581786 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.581741    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-sys\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.581940 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.581792    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-podres\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.581940 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.581798    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-lib-modules\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.581940 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.581819    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-sys\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.581940 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.581808    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33b459ea-a712-43f4-b8d6-48ee362668aa-proc\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.589137 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.589116    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l9pk\" (UniqueName: \"kubernetes.io/projected/33b459ea-a712-43f4-b8d6-48ee362668aa-kube-api-access-6l9pk\") pod \"perf-node-gather-daemonset-shrps\" (UID: \"33b459ea-a712-43f4-b8d6-48ee362668aa\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.709688 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.709623    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:43.979664 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:43.979636    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"]
Sep 16 05:30:43.982196 ip-10-0-133-117 kubenswrapper[2573]: W0916 05:30:43.982166    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod33b459ea_a712_43f4_b8d6_48ee362668aa.slice/crio-31391aaa94b850f5bdda39dcdde387c00460979d9108cd9a52ea76987f906980 WatchSource:0}: Error finding container 31391aaa94b850f5bdda39dcdde387c00460979d9108cd9a52ea76987f906980: Status 404 returned error can't find the container with id 31391aaa94b850f5bdda39dcdde387c00460979d9108cd9a52ea76987f906980
Sep 16 05:30:44.503831 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:44.503795    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps" event={"ID":"33b459ea-a712-43f4-b8d6-48ee362668aa","Type":"ContainerStarted","Data":"084187d92b9f8b1d4102b3b0714c35d1ed3715427aa4e21542daabf496ed2e2f"}
Sep 16 05:30:44.504207 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:44.503849    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps" event={"ID":"33b459ea-a712-43f4-b8d6-48ee362668aa","Type":"ContainerStarted","Data":"31391aaa94b850f5bdda39dcdde387c00460979d9108cd9a52ea76987f906980"}
Sep 16 05:30:44.504207 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:44.503884    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:44.518609 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:44.518548    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps" podStartSLOduration=1.518532749 podStartE2EDuration="1.518532749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 05:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:30:44.517709324 +0000 UTC m=+2508.183638939" watchObservedRunningTime="2026-09-16 05:30:44.518532749 +0000 UTC m=+2508.184462345"
Sep 16 05:30:44.949924 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:44.949892    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hfn7r_58152a4a-74ba-4f56-a870-99a5bae3e7af/dns/0.log"
Sep 16 05:30:44.968383 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:44.968349    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hfn7r_58152a4a-74ba-4f56-a870-99a5bae3e7af/kube-rbac-proxy/0.log"
Sep 16 05:30:45.024585 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:45.024560    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-69szj_64223cf5-19e6-4f4a-aa87-67766e0373d9/dns-node-resolver/0.log"
Sep 16 05:30:45.704631 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:45.704584    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rw649_01b6aa0b-369b-4585-bcbc-890512014439/node-ca/0.log"
Sep 16 05:30:47.011926 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:47.011892    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lm4ts_04d91007-8ea8-4099-96ca-942aafbd5768/discovery/0.log"
Sep 16 05:30:47.057120 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:47.057093    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6965d5747b-dsn8t_dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd/kube-auth-proxy/0.log"
Sep 16 05:30:48.056017 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:48.055980    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4jtn2_5d19721b-2447-4d1f-b913-df5646e2baa0/serve-healthcheck-canary/0.log"
Sep 16 05:30:48.654300 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:48.654244    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q7ss4_b7b38734-74c5-457e-9b84-32938c288a83/kube-rbac-proxy/0.log"
Sep 16 05:30:48.672641 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:48.672618    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q7ss4_b7b38734-74c5-457e-9b84-32938c288a83/exporter/0.log"
Sep 16 05:30:48.691505 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:48.691449    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q7ss4_b7b38734-74c5-457e-9b84-32938c288a83/extractor/0.log"
Sep 16 05:30:50.517548 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:50.517480    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-shrps"
Sep 16 05:30:51.304463 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:51.304435    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-c8zqs_2339d094-fc27-46a3-bd1a-8b3b738dcbd2/server/0.log"
Sep 16 05:30:51.385507 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:51.385484    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lxpdd_532d8a7e-b308-41f0-8779-351326953242/manager/0.log"
Sep 16 05:30:51.534433 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:51.534409    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756589c867-w899b_b8661495-0d52-469e-b831-f10980137a67/manager/0.log"
Sep 16 05:30:51.554024 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:51.553999    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756589c867-w899b_b8661495-0d52-469e-b831-f10980137a67/copy-manifests/0.log"
Sep 16 05:30:52.977645 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:52.976412    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-twxxl_1ddc9a3d-f768-4151-8685-7db82c15e9b9/openshift-lws-operator/0.log"
Sep 16 05:30:53.602266 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:53.602189    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lm4ts_04d91007-8ea8-4099-96ca-942aafbd5768/discovery/0.log"
Sep 16 05:30:53.623799 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:53.623766    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6965d5747b-dsn8t_dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd/kube-auth-proxy/0.log"
Sep 16 05:30:55.688825 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:55.688794    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lm4ts_04d91007-8ea8-4099-96ca-942aafbd5768/discovery/0.log"
Sep 16 05:30:55.724422 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:55.724395    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6965d5747b-dsn8t_dfa1784a-f1bd-4c5c-8dae-3dc8fcdb0dfd/kube-auth-proxy/0.log"
Sep 16 05:30:57.947898 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:57.947844    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-j8nd6_50c070c3-26e5-4761-8655-cf31cbc767df/migrator/0.log"
Sep 16 05:30:57.966051 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:57.966029    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-j8nd6_50c070c3-26e5-4761-8655-cf31cbc767df/graceful-termination/0.log"
Sep 16 05:30:59.310496 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:59.310469    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-78g6m_7b25e368-49cf-4c2e-840d-2347f6ceb400/kube-multus/0.log"
Sep 16 05:30:59.390761 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:59.390744    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tvkn_acab9663-a074-4fbb-8596-a50e5e39b878/kube-multus-additional-cni-plugins/0.log"
Sep 16 05:30:59.408892 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:59.408876    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tvkn_acab9663-a074-4fbb-8596-a50e5e39b878/egress-router-binary-copy/0.log"
Sep 16 05:30:59.427311 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:59.427269    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tvkn_acab9663-a074-4fbb-8596-a50e5e39b878/cni-plugins/0.log"
Sep 16 05:30:59.445565 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:59.445543    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tvkn_acab9663-a074-4fbb-8596-a50e5e39b878/bond-cni-plugin/0.log"
Sep 16 05:30:59.464318 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:59.464270    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tvkn_acab9663-a074-4fbb-8596-a50e5e39b878/routeoverride-cni/0.log"
Sep 16 05:30:59.483408 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:59.483385    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tvkn_acab9663-a074-4fbb-8596-a50e5e39b878/whereabouts-cni-bincopy/0.log"
Sep 16 05:30:59.502888 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:59.502872    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tvkn_acab9663-a074-4fbb-8596-a50e5e39b878/whereabouts-cni/0.log"
Sep 16 05:30:59.870982 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:59.870956    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k449w_835f4ee4-d6b0-4fb6-b6e6-86277d856c67/network-metrics-daemon/0.log"
Sep 16 05:30:59.888208 ip-10-0-133-117 kubenswrapper[2573]: I0916 05:30:59.888182    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k449w_835f4ee4-d6b0-4fb6-b6e6-86277d856c67/kube-rbac-proxy/0.log"