Sep 09 21:36:30.778637 ip-10-0-129-45 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 21:36:30.778650 ip-10-0-129-45 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 21:36:30.778658 ip-10-0-129-45 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 21:36:30.778851 ip-10-0-129-45 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 21:36:41.976582 ip-10-0-129-45 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 21:36:41.976596 ip-10-0-129-45 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f7c0b1e26ac648909fe7c55e696cfc0b --
Sep 09 21:38:26.996092 ip-10-0-129-45 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 21:38:27.379818 ip-10-0-129-45 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:38:27.379818 ip-10-0-129-45 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 21:38:27.379818 ip-10-0-129-45 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:38:27.379818 ip-10-0-129-45 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 21:38:27.379818 ip-10-0-129-45 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:38:27.382337 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.382079    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 21:38:27.387057 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387042    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:27.387057 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387057    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387060    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387063    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387066    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387069    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387071    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387074    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387076    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387079    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387084    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387087    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387090    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387093    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387095    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387098    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387100    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387103    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387106    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387108    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387111    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:27.387125 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387114    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387117    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387119    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387123    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387127    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387133    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387136    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387139    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387143    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387146    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387149    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387152    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387155    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387158    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387160    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387163    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387166    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387168    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387171    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:27.387587 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387173    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387176    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387178    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387181    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387184    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387187    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387189    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387191    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387194    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387196    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387199    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387202    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387204    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387206    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387209    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387212    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387215    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387217    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387220    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387222    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:27.388054 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387225    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387227    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387229    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387232    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387235    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387238    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387241    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387243    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387246    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387248    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387251    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387253    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387256    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387258    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387261    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387263    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387290    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387293    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387296    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387299    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:27.388540 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387302    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387305    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387308    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387311    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387314    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387316    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387319    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387322    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387325    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387328    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387330    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387333    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387335    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387338    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387340    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387343    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387345    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387348    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.387350    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388054    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:27.389036 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388062    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388065    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388068    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388071    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388074    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388077    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388081    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388084    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388087    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388089    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388093    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388096    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388100    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388105    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388108    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388111    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388114    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388116    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388120    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388124    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:27.389518 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388127    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388130    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388133    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388135    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388138    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388141    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388143    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388146    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388149    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388152    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388154    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388157    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388160    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388162    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388165    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388167    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388169    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388172    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388174    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388177    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:27.390028 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388179    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388182    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388184    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388187    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388190    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388193    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388196    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388198    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388201    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388203    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388206    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388209    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388211    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388214    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388216    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388219    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388221    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388224    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388226    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388229    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:27.390508 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388232    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388234    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388236    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388239    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388242    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388245    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388247    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388250    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388252    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388254    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388257    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388260    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388263    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388267    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388270    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388273    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388275    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388278    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388280    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:27.391027 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388283    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388286    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388288    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388290    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388294    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388296    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388299    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388301    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388303    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388306    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388308    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388312    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388315    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388318    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388321    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388323    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388326    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388328    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388331    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:27.391490 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388723    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388728    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388731    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388734    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388736    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388739    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388742    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388744    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388747    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388750    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388752    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388754    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388757    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388760    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388763    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388779    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388782    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388786    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388790    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:27.391961 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388793    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388795    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388798    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388800    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388803    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388806    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388808    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388811    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388814    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388816    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388819    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388821    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388824    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388827    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388829    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388831    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388834    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388836    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388843    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:27.392461 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388846    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388850    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388854    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388857    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388860    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388863    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388866    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388868    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388872    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388875    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388878    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388880    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388883    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388885    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388888    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388890    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388893    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388895    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388898    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:27.392937 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388900    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388902    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388905    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388907    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388910    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388912    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388915    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388918    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388920    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388923    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388926    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388928    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388932    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388935    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388937    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388941    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388944    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388946    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388949    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388951    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:27.393491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388954    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388956    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388959    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388962    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388964    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388967    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388971    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388974    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388977    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388980    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388982    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388985    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388988    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388991    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388993    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388996    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.388999    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389001    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389004    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389007    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:27.394019 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389010    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389013    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389101    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389147    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389155    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389160    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389164    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389167    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389171    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389176    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389179    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389182    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389185    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389188    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389191    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389195    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389198    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389201    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389204    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389207    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389211    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389214    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389217    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389220    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 21:38:27.394496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389223    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389227    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389230    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389233    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389236    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389239    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389242    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389245    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389248    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389251    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389255    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389258    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389262    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389265    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389268    2563 flags.go:64] FLAG: --enable-server="true"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389271    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389280    2563 flags.go:64] FLAG: --event-burst="100"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389283    2563 flags.go:64] FLAG: --event-qps="50"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389285    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389288    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389291    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389295    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389297    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389300    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389304    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 09 21:38:27.395078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389307    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389310    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389312    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389315    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389318    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389321    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389323    2563 flags.go:64] FLAG: --feature-gates=""
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389327    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389330    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389333    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389336    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389339    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389342    2563 flags.go:64] FLAG: --help="false"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389344    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389347    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389350    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389353    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389356    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389359    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389363    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389366    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389368    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389371    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 21:38:27.395700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389376    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389379    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389381    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389384    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389387    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389390    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389393    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389395    2563 flags.go:64] FLAG: --lock-file=""
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389398    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389402    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389405    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389410    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389413    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389416    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389418    2563 flags.go:64] FLAG: --logging-format="text"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389421    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389424    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389426    2563 flags.go:64] FLAG: --manifest-url=""
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389429    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389433    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389436    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389440    2563 flags.go:64] FLAG: --max-pods="110"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389443    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389446    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389448    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 21:38:27.396293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389451    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389454    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389457    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389460    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389467    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389470    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389473    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389477    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389480    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389486    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389489    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389492    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389495    2563 flags.go:64] FLAG: --port="10250"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389498    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389501    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-044f5eb5400b76cca"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389504    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389506    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389511    2563 flags.go:64] FLAG: --register-node="true"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389514    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389517    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389520    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389523    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389526    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389530    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389533    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 21:38:27.396921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389536    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389539    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389542    2563 flags.go:64] FLAG: --runonce="false"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389545    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389548    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389551    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389554    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389556    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389560    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389563    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389566    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389570    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389572    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389575    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389579    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389581    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389584    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389587    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389592    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389595    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389597    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389601    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389604    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389606    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389609    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 21:38:27.397655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389612    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389615    2563 flags.go:64] FLAG: --v="2"
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389619    2563 flags.go:64] FLAG: --version="false"
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389623    2563 flags.go:64] FLAG: --vmodule=""
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389627    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.389630    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389739    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389743    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389746    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389749    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389752    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389754    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389757    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389759    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389762    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389779    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389782    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389785    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389788    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389791    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389793    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:27.398352 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389798    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389801    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389803    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389806    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389808    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389811    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389813    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389816    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389819    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389821    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389823    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389826    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389829    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389831    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389834    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389836    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389839    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389841    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389844    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389846    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:27.398969 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389849    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389851    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389853    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389856    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389858    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389861    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389864    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389866    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389870    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389874    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389877    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389881    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389885    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389888    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389890    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389893    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389895    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389898    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389900    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:27.399530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389903    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389905    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389909    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389912    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389915    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389918    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389921    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389924    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389926    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389929    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389931    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389933    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389936    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389938    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389941    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389943    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389946    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389948    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389951    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389954    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:27.400071 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389956    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389959    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389961    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389964    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389966    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389970    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389972    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389974    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389977    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389988    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389991    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389993    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389996    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.389998    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390001    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390004    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390006    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390009    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390011    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390014    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:27.400641 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390017    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390019    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390021    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390024    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.390026    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.390900    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.398619    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.398634    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.398650    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.398660    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398715    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398719    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398723    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398726    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398728    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:27.401228 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398731    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398734    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398736    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398739    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398741    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398744    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398746    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398749    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398752    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398754    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398757    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398759    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398762    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398784    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398789    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398792    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398795    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398797    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398800    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398802    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:27.401620 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398805    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398808    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398811    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398813    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398817    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398820    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398822    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398825    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398827    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398830    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398832    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398835    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398837    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398840    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398842    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398845    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398848    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398850    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398852    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:27.402110 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398855    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398857    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398860    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398862    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398865    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398867    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398869    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398872    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398874    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398877    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398879    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398882    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398884    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398887    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398889    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398892    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398894    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398897    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398900    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398903    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:27.402578 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398905    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398908    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398910    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398912    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398915    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398917    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398920    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398922    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398925    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398927    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398930    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398932    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398935    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398937    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398939    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398942    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398946    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398949    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398952    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:27.403080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398955    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398959    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398962    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398966    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398969    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398972    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398975    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398979    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398982    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398984    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398988    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398990    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398993    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398996    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.398998    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:27.403534 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399000    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.399005    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399120    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399125    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399128    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399131    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399134    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399137    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399140    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399143    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399145    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399148    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399150    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399153    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399156    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:27.403920 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399158    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399161    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399163    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399166    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399168    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399172    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399174    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399176    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399180    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399184    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399187    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399190    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399192    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399195    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399197    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399200    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399203    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399205    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399208    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399210    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:27.404302 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399213    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399215    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399218    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399220    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399222    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399225    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399227    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399230    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399232    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399234    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399237    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399239    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399242    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399244    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399247    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399249    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399251    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399254    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399256    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399259    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:27.404786 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399261    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399264    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399266    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399269    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399271    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399274    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399276    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399279    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399282    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399285    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399287    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399289    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399292    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399294    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399297    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399299    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399301    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399304    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399306    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:27.405264 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399309    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399311    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399313    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399316    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399318    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399320    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399323    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399325    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399327    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399330    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399332    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399335    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399337    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399340    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399342    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399345    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399347    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399349    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399352    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399354    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:27.405731 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399357    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399359    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399362    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399366    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399369    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399372    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:27.399374    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.399379    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.399488    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.401688    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.402750    2563 server.go:1019] "Starting client certificate rotation"
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.402859    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.402888    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 21:38:27.406224 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.405738    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 21:38:27.407038 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.407019    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 21:38:27.420238 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.420211    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 09 21:38:27.423504 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.423485    2563 log.go:25] "Validated CRI v1 image API"
Sep 09 21:38:27.424337 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.424320    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 21:38:27.428121 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.428103    2563 fs.go:135] Filesystem UUIDs: map[0ce940e1-4e6d-40ab-bd05-e6623d4a7d97:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9392c1bb-9e61-42e1-82d5-e65dcc6f8bc4:/dev/nvme0n1p4]
Sep 09 21:38:27.428176 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.428122    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 21:38:27.433927 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.433813    2563 manager.go:216] Machine: {Timestamp:2026-09-09 21:38:27.432342088 +0000 UTC m=+0.329008086 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100627 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269c317bec0901f17950bdf394cc64 SystemUUID:ec269c31-7bec-0901-f179-50bdf394cc64 BootID:f7c0b1e2-6ac6-4890-9fe7-c55e696cfc0b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d5:8a:3b:07:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d5:8a:3b:07:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:09:01:d4:08:bc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 21:38:27.433927 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.433920    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 21:38:27.434059 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.433996    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 21:38:27.436103 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.436083    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 21:38:27.438230 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.438200    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 21:38:27.438397 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.438235    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 21:38:27.439042 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.439030    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 21:38:27.439102 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.439048    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 21:38:27.439102 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.439058    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 21:38:27.439102 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.439081    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 21:38:27.440647 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.440634    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 21:38:27.444657 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.444644    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 21:38:27.444826 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.444815    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 21:38:27.444894 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.444852    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 21:38:27.444894 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.444866    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 21:38:27.444894 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.444881    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 21:38:27.445009 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.444897    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 09 21:38:27.445009 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.444908    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 21:38:27.445296 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.445281    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 21:38:27.445418 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.445304    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 21:38:27.449800 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.449784    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 21:38:27.452644 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.452631    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 21:38:27.452708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.452660    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 21:38:27.452864 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.452851    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 21:38:27.452896 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.452871    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 21:38:27.452896 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.452877    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 21:38:27.452896 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.452883    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 21:38:27.452896 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.452891    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 21:38:27.452999 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.452898    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 21:38:27.453468 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.453455    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 21:38:27.453496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.453471    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 21:38:27.453496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.453481    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 21:38:27.453496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.453487    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 21:38:27.453496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.453496    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 21:38:27.453598 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.453504    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 21:38:27.454758 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.454745    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 21:38:27.454758 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.454756    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 21:38:27.455831 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.455791    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 21:38:27.455899 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.455859    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 21:38:27.458727 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.458556    2563 server.go:1268] "Started kubelet"
Sep 09 21:38:27.458812 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.458647    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 21:38:27.459506 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.459406    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 21:38:27.459601 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.459548    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 21:38:27.460728 ip-10-0-129-45 systemd[1]: Started Kubernetes Kubelet.
Sep 09 21:38:27.461025 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.460855    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 21:38:27.462147 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.462133    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 21:38:27.463251 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.463215    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 21:38:27.465300 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.465282    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 21:38:27.465378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.465299    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 21:38:27.465623 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.465578    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:27.467001 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.466975    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 21:38:27.467423 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.467405    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 21:38:27.467423 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.467424    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 21:38:27.467547 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.467454    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 21:38:27.467597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.467554    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 21:38:27.467597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.467563    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 21:38:27.467759 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.467732    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 21:38:27.467759 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.467752    2563 factory.go:55] Registering systemd factory
Sep 09 21:38:27.467759 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.467762    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 09 21:38:27.468093 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.468076    2563 factory.go:153] Registering CRI-O factory
Sep 09 21:38:27.468093 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.468090    2563 factory.go:223] Registration of the crio container factory successfully
Sep 09 21:38:27.468218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.468107    2563 factory.go:103] Registering Raw factory
Sep 09 21:38:27.468218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.468120    2563 manager.go:1196] Started watching for new ooms in manager
Sep 09 21:38:27.468594 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.468581    2563 manager.go:349] Starting recovery of all containers
Sep 09 21:38:27.472590 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.467525    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-45.ec2.internal.18d3c4d7a4d31279  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-45.ec2.internal,UID:ip-10-0-129-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-45.ec2.internal,},FirstTimestamp:2026-09-09 21:38:27.458527865 +0000 UTC m=+0.355193864,LastTimestamp:2026-09-09 21:38:27.458527865 +0000 UTC m=+0.355193864,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-45.ec2.internal,}"
Sep 09 21:38:27.476922 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.476015    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 21:38:27.476922 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.476122    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 21:38:27.482012 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.481992    2563 manager.go:354] Recovery completed
Sep 09 21:38:27.483820 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.483755    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b6xdz"
Sep 09 21:38:27.487170 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.487157    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:27.488952 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.488938    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:27.488996 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.488965    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:27.488996 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.488979    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:27.489562 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.489547    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 21:38:27.489562 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.489554    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:27.489562 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.489560    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 21:38:27.489676 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.489574    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 21:38:27.490242 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.490229    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:27.490281 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.490255    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:27.490281 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.490269    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:27.490957 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.490870    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-45.ec2.internal.18d3c4d7a6a3527c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-45.ec2.internal,UID:ip-10-0-129-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-45.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-45.ec2.internal,},FirstTimestamp:2026-09-09 21:38:27.488952956 +0000 UTC m=+0.385618955,LastTimestamp:2026-09-09 21:38:27.488952956 +0000 UTC m=+0.385618955,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-45.ec2.internal,}"
Sep 09 21:38:27.491040 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.490940    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b6xdz"
Sep 09 21:38:27.491659 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.491646    2563 policy_none.go:49] "None policy: Start"
Sep 09 21:38:27.491701 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.491663    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 21:38:27.491701 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.491673    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 21:38:27.493945 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.493933    2563 policy_none.go:47] "Start"
Sep 09 21:38:27.548534 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.530006    2563 manager.go:337] "Starting Device Plugin manager"
Sep 09 21:38:27.548534 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.530043    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 21:38:27.548534 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.530058    2563 server.go:87] "Starting device plugin registration server"
Sep 09 21:38:27.548534 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.530371    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 21:38:27.548534 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.530384    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 21:38:27.548534 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.530480    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 21:38:27.548534 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.530553    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 21:38:27.548534 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.530564    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 21:38:27.548534 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.531105    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 21:38:27.548534 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.531143    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:27.597780 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.597734    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 21:38:27.599132 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.599114    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 21:38:27.599199 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.599177    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 21:38:27.599292 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.599207    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 21:38:27.599292 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.599254    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 21:38:27.602791 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.602759    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:27.631504 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.631444    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:27.632302 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.632284    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:27.632392 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.632314    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:27.632392 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.632323    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:27.632392 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.632349    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.638589 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.638576    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.638633 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.638597    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-45.ec2.internal\": node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:27.661654 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.661625    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:27.700265 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.700088    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"]
Sep 09 21:38:27.701845 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.700837    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:27.702734 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.702713    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:27.702853 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.702746    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:27.702853 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.702760    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:27.705106 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.705094    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:27.705253 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.705235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.705341 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.705265    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:27.705803 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.705790    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:27.705857 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.705814    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:27.705857 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.705827    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:27.706307 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.706293    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:27.706349 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.706322    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:27.706349 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.706333    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:27.708474 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.708460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.708524 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.708485    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:27.709169 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.709150    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:27.709252 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.709181    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:27.709252 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.709195    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:27.738409 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.738391    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-45.ec2.internal\" not found" node="ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.742652 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.742636    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-45.ec2.internal\" not found" node="ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.762205 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.762186    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:27.769620 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.769601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.769688 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.769663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.769730 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.769699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1034d32653d0ee36541005678f807d7f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"1034d32653d0ee36541005678f807d7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.862573 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.862534    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:27.870010 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.869982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1034d32653d0ee36541005678f807d7f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"1034d32653d0ee36541005678f807d7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.870089 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.870015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.870089 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.870033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.870089 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.870086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.870202 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.870091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1034d32653d0ee36541005678f807d7f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"1034d32653d0ee36541005678f807d7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.870202 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:27.870126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:27.963509 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:27.963428    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:28.041054 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.041022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:28.045747 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.045728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:28.064432 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.064407    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:28.165022 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.164976    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:28.265583 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.265498    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:28.366083 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.366048    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 21:38:28.402543 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.402507    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 21:38:28.403000 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.402671    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 21:38:28.417462 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.417441    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:28.445549 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.445526    2563 apiserver.go:52] "Watching apiserver"
Sep 09 21:38:28.454290 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.454270    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 21:38:28.455376 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.455357    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-nd9gr","kube-system/konnectivity-agent-5npzg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz","openshift-cluster-node-tuning-operator/tuned-kbhzz","openshift-dns/node-resolver-68g9z","openshift-image-registry/node-ca-tlfbl","openshift-multus/network-metrics-daemon-vsdv8","openshift-network-diagnostics/network-check-target-2jq5r","openshift-multus/multus-additional-cni-plugins-wnfn6","openshift-multus/multus-krpdt","openshift-network-operator/iptables-alerter-pj8vm","openshift-ovn-kubernetes/ovnkube-node-zchqg"]
Sep 09 21:38:28.457856 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.457843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:28.457923 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.457907    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:28.459998 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.459982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:28.462118 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.462101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.462665 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.462648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 21:38:28.462735 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.462723    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dsvkz\""
Sep 09 21:38:28.462856 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.462844    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 21:38:28.464088 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.464063    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hczq7\""
Sep 09 21:38:28.464185 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.464153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 21:38:28.464286 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.464272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.464436 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.464420    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 21:38:28.464525 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.464508    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 21:38:28.465815 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.465801    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 21:38:28.465889 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.465804    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:28.466375 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.466353    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:38:28.466497 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.466476    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nvrhq\""
Sep 09 21:38:28.466603 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.466509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.466603 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.466506    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 21:38:28.468590 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.468575    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.468789 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.468749    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 21:38:28.468928 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.468913    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 21:38:28.469008 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.468995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sllv9\""
Sep 09 21:38:28.470756 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.470740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:28.470860 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.470795    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 21:38:28.470860 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.470815    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:28.471012 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.470997    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pwz2d\""
Sep 09 21:38:28.471050 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.471009    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 21:38:28.471050 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.471000    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 21:38:28.472824 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.472808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:28.472884 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.472857    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:28.474210 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-var-lib-kubelet\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.474273 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9b64455-a274-4e65-91b6-849f97952b2a-agent-certs\") pod \"konnectivity-agent-5npzg\" (UID: \"c9b64455-a274-4e65-91b6-849f97952b2a\") " pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:28.474423 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9b64455-a274-4e65-91b6-849f97952b2a-konnectivity-ca\") pod \"konnectivity-agent-5npzg\" (UID: \"c9b64455-a274-4e65-91b6-849f97952b2a\") " pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:28.474476 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-sysctl-conf\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.474525 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-lib-modules\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.474525 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-tuned\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.474613 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/715cecb6-2493-4cd4-84b8-6eb84e62fb1b-serviceca\") pod \"node-ca-tlfbl\" (UID: \"715cecb6-2493-4cd4-84b8-6eb84e62fb1b\") " pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.474613 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-run\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.474613 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-kubelet-config\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:28.474741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.474741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-socket-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.474741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-modprobe-d\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.474741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-systemd\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.474936 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.474747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7198ae4-9d2d-41cb-892c-90e8f729c20e-hosts-file\") pod \"node-resolver-68g9z\" (UID: \"d7198ae4-9d2d-41cb-892c-90e8f729c20e\") " pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.475578 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-device-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.475664 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-etc-selinux\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.475664 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c6210d3-1793-4304-9e56-c7c915be4fda-tmp\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.475787 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddq9\" (UniqueName: \"kubernetes.io/projected/715cecb6-2493-4cd4-84b8-6eb84e62fb1b-kube-api-access-lddq9\") pod \"node-ca-tlfbl\" (UID: \"715cecb6-2493-4cd4-84b8-6eb84e62fb1b\") " pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.475787 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:28.475787 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-host\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.475787 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-sys\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.475982 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7198ae4-9d2d-41cb-892c-90e8f729c20e-tmp-dir\") pod \"node-resolver-68g9z\" (UID: \"d7198ae4-9d2d-41cb-892c-90e8f729c20e\") " pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.475982 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/715cecb6-2493-4cd4-84b8-6eb84e62fb1b-host\") pod \"node-ca-tlfbl\" (UID: \"715cecb6-2493-4cd4-84b8-6eb84e62fb1b\") " pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.475982 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-registration-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.475982 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-kubernetes\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.475982 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9npn\" (UniqueName: \"kubernetes.io/projected/d7198ae4-9d2d-41cb-892c-90e8f729c20e-kube-api-access-r9npn\") pod \"node-resolver-68g9z\" (UID: \"d7198ae4-9d2d-41cb-892c-90e8f729c20e\") " pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.475982 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475916    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 21:38:28.475982 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-sys-fs\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.475982 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.475978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbnhl\" (UniqueName: \"kubernetes.io/projected/6eaafd9b-34bd-42ea-9642-2c99c8204743-kube-api-access-bbnhl\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.476335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.476004    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 21:38:28.476335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.476006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzhr\" (UniqueName: \"kubernetes.io/projected/3c6210d3-1793-4304-9e56-c7c915be4fda-kube-api-access-zmzhr\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.476335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.476036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-dbus\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:28.476335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.476058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-sysconfig\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.476335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.476086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-sysctl-d\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.476568 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.476372    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.477887 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.477865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.478605 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.478588    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 21:38:28.479237 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.479217    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 21:38:28.479342 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.479318    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 21:38:28.479539 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.479508    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 21:38:28.479660 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.479553    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jk262\""
Sep 09 21:38:28.480335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.479813    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 21:38:28.480335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.480219    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 21:38:28.480335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.480231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.480498 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.480412    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2qpx5\""
Sep 09 21:38:28.481350 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.481333    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 21:38:28.482548 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.482519    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 21:38:28.482639 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.482550    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"]
Sep 09 21:38:28.482639 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.482567    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h5c9m\""
Sep 09 21:38:28.482744 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.482640    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:38:28.482744 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.482651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.482883 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.482753    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 21:38:28.484721 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.484704    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 21:38:28.484899 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.484884    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 21:38:28.484947 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.484924    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 21:38:28.485183 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.485164    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 21:38:28.485183 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.485176    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 21:38:28.485308 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.485186    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t2scw\""
Sep 09 21:38:28.485308 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.485204    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 21:38:28.490153 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.490135    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"]
Sep 09 21:38:28.490229 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.490169    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 21:38:28.493576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.493528    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 21:33:27 +0000 UTC" deadline="2028-02-05 16:09:52.928964758 +0000 UTC"
Sep 09 21:38:28.493576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.493574    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12330h31m24.435393919s"
Sep 09 21:38:28.507884 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.507867    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lzksk"
Sep 09 21:38:28.516460 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.516421    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lzksk"
Sep 09 21:38:28.567782 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.567754    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 21:38:28.576280 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da6de3d9-7174-44f8-9773-fc7f5b353bd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.576372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-run-netns\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.576372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-var-lib-cni-bin\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.576372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-daemon-config\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.576372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.576523 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.576523 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-modprobe-d\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.576523 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7198ae4-9d2d-41cb-892c-90e8f729c20e-hosts-file\") pod \"node-resolver-68g9z\" (UID: \"d7198ae4-9d2d-41cb-892c-90e8f729c20e\") " pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.576681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-var-lib-openvswitch\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.576681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-modprobe-d\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.576681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7198ae4-9d2d-41cb-892c-90e8f729c20e-hosts-file\") pod \"node-resolver-68g9z\" (UID: \"d7198ae4-9d2d-41cb-892c-90e8f729c20e\") " pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.576681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-device-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.576681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-device-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.576681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvs4w\" (UniqueName: \"kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w\") pod \"network-check-target-2jq5r\" (UID: \"0e74454c-5e31-4943-881c-02cabbbd5bfc\") " pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:28.576681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6de3d9-7174-44f8-9773-fc7f5b353bd8-system-cni-dir\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7198ae4-9d2d-41cb-892c-90e8f729c20e-tmp-dir\") pod \"node-resolver-68g9z\" (UID: \"d7198ae4-9d2d-41cb-892c-90e8f729c20e\") " pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-var-lib-cni-multus\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-systemd-units\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-run-systemd\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-run-ovn\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-kubernetes\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/da6de3d9-7174-44f8-9773-fc7f5b353bd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-conf-dir\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-run-ovn-kubernetes\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/773f5b77-c88f-4e79-8104-fd4f9bc8c196-ovnkube-config\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/773f5b77-c88f-4e79-8104-fd4f9bc8c196-ovn-node-metrics-cert\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-sys-fs\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-kubernetes\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmp7l\" (UniqueName: \"kubernetes.io/projected/18bb7939-b868-42f1-9c76-e12060047d00-kube-api-access-mmp7l\") pod \"iptables-alerter-pj8vm\" (UID: \"18bb7939-b868-42f1-9c76-e12060047d00\") " pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.576955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-dbus\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.576989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-sysconfig\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577004    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-sys-fs\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-sysctl-d\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7198ae4-9d2d-41cb-892c-90e8f729c20e-tmp-dir\") pod \"node-resolver-68g9z\" (UID: \"d7198ae4-9d2d-41cb-892c-90e8f729c20e\") " pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-dbus\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-var-lib-kubelet\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-sysconfig\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577073    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-system-cni-dir\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5aca247b-df86-41d9-9b9d-c9e302f8df71-cni-binary-copy\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-var-lib-kubelet\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-sysctl-d\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-node-log\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-log-socket\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/da6de3d9-7174-44f8-9773-fc7f5b353bd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-run-k8s-cni-cncf-io\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-sysctl-conf\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.577492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-lib-modules\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/715cecb6-2493-4cd4-84b8-6eb84e62fb1b-serviceca\") pod \"node-ca-tlfbl\" (UID: \"715cecb6-2493-4cd4-84b8-6eb84e62fb1b\") " pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da6de3d9-7174-44f8-9773-fc7f5b353bd8-cnibin\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da6de3d9-7174-44f8-9773-fc7f5b353bd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-cni-netd\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddq9\" (UniqueName: \"kubernetes.io/projected/715cecb6-2493-4cd4-84b8-6eb84e62fb1b-kube-api-access-lddq9\") pod \"node-ca-tlfbl\" (UID: \"715cecb6-2493-4cd4-84b8-6eb84e62fb1b\") " pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-cni-dir\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-run\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-kubelet-config\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-socket-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-lib-modules\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-systemd\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-run\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-etc-kubernetes\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-kubelet-config\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-etc-selinux\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-sysctl-conf\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.578148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-systemd\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c6210d3-1793-4304-9e56-c7c915be4fda-tmp\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6t8r\" (UniqueName: \"kubernetes.io/projected/5aca247b-df86-41d9-9b9d-c9e302f8df71-kube-api-access-w6t8r\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-etc-selinux\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-socket-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/773f5b77-c88f-4e79-8104-fd4f9bc8c196-env-overrides\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6jm\" (UniqueName: \"kubernetes.io/projected/f9a1488a-6bce-4e7b-846d-37825f27b619-kube-api-access-2h6jm\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18bb7939-b868-42f1-9c76-e12060047d00-host-slash\") pod \"iptables-alerter-pj8vm\" (UID: \"18bb7939-b868-42f1-9c76-e12060047d00\") " pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-host\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/715cecb6-2493-4cd4-84b8-6eb84e62fb1b-serviceca\") pod \"node-ca-tlfbl\" (UID: \"715cecb6-2493-4cd4-84b8-6eb84e62fb1b\") " pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-kubelet\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-host\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.577726    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577761    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.577809    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret podName:6271f963-afc0-421e-8d4f-cbe5eedbc4d8 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:29.077793734 +0000 UTC m=+1.974459723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret") pod "global-pull-secret-syncer-nd9gr" (UID: "6271f963-afc0-421e-8d4f-cbe5eedbc4d8") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-etc-openvswitch\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.578708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799n6\" (UniqueName: \"kubernetes.io/projected/773f5b77-c88f-4e79-8104-fd4f9bc8c196-kube-api-access-799n6\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-sys\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/715cecb6-2493-4cd4-84b8-6eb84e62fb1b-host\") pod \"node-ca-tlfbl\" (UID: \"715cecb6-2493-4cd4-84b8-6eb84e62fb1b\") " pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-cnibin\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-hostroot\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c6210d3-1793-4304-9e56-c7c915be4fda-sys\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/715cecb6-2493-4cd4-84b8-6eb84e62fb1b-host\") pod \"node-ca-tlfbl\" (UID: \"715cecb6-2493-4cd4-84b8-6eb84e62fb1b\") " pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.577984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-registration-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9npn\" (UniqueName: \"kubernetes.io/projected/d7198ae4-9d2d-41cb-892c-90e8f729c20e-kube-api-access-r9npn\") pod \"node-resolver-68g9z\" (UID: \"d7198ae4-9d2d-41cb-892c-90e8f729c20e\") " pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6eaafd9b-34bd-42ea-9642-2c99c8204743-registration-dir\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-socket-dir-parent\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-var-lib-kubelet\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-slash\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/18bb7939-b868-42f1-9c76-e12060047d00-iptables-alerter-script\") pod \"iptables-alerter-pj8vm\" (UID: \"18bb7939-b868-42f1-9c76-e12060047d00\") " pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbnhl\" (UniqueName: \"kubernetes.io/projected/6eaafd9b-34bd-42ea-9642-2c99c8204743-kube-api-access-bbnhl\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzhr\" (UniqueName: \"kubernetes.io/projected/3c6210d3-1793-4304-9e56-c7c915be4fda-kube-api-access-zmzhr\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-run-multus-certs\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.579246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-run-netns\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.579718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-run-openvswitch\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.579718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-cni-bin\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.579718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4pp\" (UniqueName: \"kubernetes.io/projected/da6de3d9-7174-44f8-9773-fc7f5b353bd8-kube-api-access-db4pp\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.579718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.579718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/773f5b77-c88f-4e79-8104-fd4f9bc8c196-ovnkube-script-lib\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.579718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:28.579718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9b64455-a274-4e65-91b6-849f97952b2a-agent-certs\") pod \"konnectivity-agent-5npzg\" (UID: \"c9b64455-a274-4e65-91b6-849f97952b2a\") " pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:28.579718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9b64455-a274-4e65-91b6-849f97952b2a-konnectivity-ca\") pod \"konnectivity-agent-5npzg\" (UID: \"c9b64455-a274-4e65-91b6-849f97952b2a\") " pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:28.579718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-tuned\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.579718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.578850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9b64455-a274-4e65-91b6-849f97952b2a-konnectivity-ca\") pod \"konnectivity-agent-5npzg\" (UID: \"c9b64455-a274-4e65-91b6-849f97952b2a\") " pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:28.581076 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.581062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c6210d3-1793-4304-9e56-c7c915be4fda-tmp\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.581127 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.581100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c6210d3-1793-4304-9e56-c7c915be4fda-etc-tuned\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.581298 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.581283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9b64455-a274-4e65-91b6-849f97952b2a-agent-certs\") pod \"konnectivity-agent-5npzg\" (UID: \"c9b64455-a274-4e65-91b6-849f97952b2a\") " pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:28.590211 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.590190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzhr\" (UniqueName: \"kubernetes.io/projected/3c6210d3-1793-4304-9e56-c7c915be4fda-kube-api-access-zmzhr\") pod \"tuned-kbhzz\" (UID: \"3c6210d3-1793-4304-9e56-c7c915be4fda\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.590211 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.590206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9npn\" (UniqueName: \"kubernetes.io/projected/d7198ae4-9d2d-41cb-892c-90e8f729c20e-kube-api-access-r9npn\") pod \"node-resolver-68g9z\" (UID: \"d7198ae4-9d2d-41cb-892c-90e8f729c20e\") " pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.590346 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.590262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddq9\" (UniqueName: \"kubernetes.io/projected/715cecb6-2493-4cd4-84b8-6eb84e62fb1b-kube-api-access-lddq9\") pod \"node-ca-tlfbl\" (UID: \"715cecb6-2493-4cd4-84b8-6eb84e62fb1b\") " pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.590717 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.590694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbnhl\" (UniqueName: \"kubernetes.io/projected/6eaafd9b-34bd-42ea-9642-2c99c8204743-kube-api-access-bbnhl\") pod \"aws-ebs-csi-driver-node-lzjcz\" (UID: \"6eaafd9b-34bd-42ea-9642-2c99c8204743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.676584 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.676556    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:28.679016 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.678993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-run-ovn-kubernetes\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.679131 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/773f5b77-c88f-4e79-8104-fd4f9bc8c196-ovnkube-config\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.679131 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/773f5b77-c88f-4e79-8104-fd4f9bc8c196-ovn-node-metrics-cert\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.679131 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmp7l\" (UniqueName: \"kubernetes.io/projected/18bb7939-b868-42f1-9c76-e12060047d00-kube-api-access-mmp7l\") pod \"iptables-alerter-pj8vm\" (UID: \"18bb7939-b868-42f1-9c76-e12060047d00\") " pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.679131 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-run-ovn-kubernetes\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.679131 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-system-cni-dir\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.679131 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5aca247b-df86-41d9-9b9d-c9e302f8df71-cni-binary-copy\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-node-log\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-log-socket\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/da6de3d9-7174-44f8-9773-fc7f5b353bd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-run-k8s-cni-cncf-io\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-system-cni-dir\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da6de3d9-7174-44f8-9773-fc7f5b353bd8-cnibin\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da6de3d9-7174-44f8-9773-fc7f5b353bd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-cni-netd\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-cni-dir\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-etc-kubernetes\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6t8r\" (UniqueName: \"kubernetes.io/projected/5aca247b-df86-41d9-9b9d-c9e302f8df71-kube-api-access-w6t8r\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da6de3d9-7174-44f8-9773-fc7f5b353bd8-cnibin\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-log-socket\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/773f5b77-c88f-4e79-8104-fd4f9bc8c196-env-overrides\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.679429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-etc-kubernetes\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6jm\" (UniqueName: \"kubernetes.io/projected/f9a1488a-6bce-4e7b-846d-37825f27b619-kube-api-access-2h6jm\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18bb7939-b868-42f1-9c76-e12060047d00-host-slash\") pod \"iptables-alerter-pj8vm\" (UID: \"18bb7939-b868-42f1-9c76-e12060047d00\") " pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-kubelet\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-etc-openvswitch\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799n6\" (UniqueName: \"kubernetes.io/projected/773f5b77-c88f-4e79-8104-fd4f9bc8c196-kube-api-access-799n6\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-cnibin\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-hostroot\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-socket-dir-parent\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-var-lib-kubelet\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-slash\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/18bb7939-b868-42f1-9c76-e12060047d00-iptables-alerter-script\") pod \"iptables-alerter-pj8vm\" (UID: \"18bb7939-b868-42f1-9c76-e12060047d00\") " pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-run-multus-certs\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-run-netns\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-run-openvswitch\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/773f5b77-c88f-4e79-8104-fd4f9bc8c196-env-overrides\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-cni-bin\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.680226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-node-log\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4pp\" (UniqueName: \"kubernetes.io/projected/da6de3d9-7174-44f8-9773-fc7f5b353bd8-kube-api-access-db4pp\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da6de3d9-7174-44f8-9773-fc7f5b353bd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/773f5b77-c88f-4e79-8104-fd4f9bc8c196-ovnkube-script-lib\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/773f5b77-c88f-4e79-8104-fd4f9bc8c196-ovnkube-config\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-cni-netd\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da6de3d9-7174-44f8-9773-fc7f5b353bd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-socket-dir-parent\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-cnibin\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-run-netns\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-etc-openvswitch\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-run-k8s-cni-cncf-io\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.679948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/da6de3d9-7174-44f8-9773-fc7f5b353bd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-cni-dir\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-var-lib-cni-bin\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5aca247b-df86-41d9-9b9d-c9e302f8df71-cni-binary-copy\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-kubelet\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-run-netns\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-daemon-config\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-var-lib-cni-bin\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-var-lib-kubelet\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18bb7939-b868-42f1-9c76-e12060047d00-host-slash\") pod \"iptables-alerter-pj8vm\" (UID: \"18bb7939-b868-42f1-9c76-e12060047d00\") " pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-slash\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-var-lib-openvswitch\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-run-openvswitch\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da6de3d9-7174-44f8-9773-fc7f5b353bd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-cni-bin\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-var-lib-openvswitch\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs4w\" (UniqueName: \"kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w\") pod \"network-check-target-2jq5r\" (UID: \"0e74454c-5e31-4943-881c-02cabbbd5bfc\") " pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-run-multus-certs\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.680314    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-host-run-netns\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.681907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-hostroot\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6de3d9-7174-44f8-9773-fc7f5b353bd8-system-cni-dir\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.680563    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs podName:f9a1488a-6bce-4e7b-846d-37825f27b619 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:29.180513359 +0000 UTC m=+2.077179347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs") pod "network-metrics-daemon-vsdv8" (UID: "f9a1488a-6bce-4e7b-846d-37825f27b619") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6de3d9-7174-44f8-9773-fc7f5b353bd8-system-cni-dir\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-var-lib-cni-multus\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-systemd-units\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-daemon-config\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-run-systemd\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-run-ovn\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-host-var-lib-cni-multus\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/da6de3d9-7174-44f8-9773-fc7f5b353bd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-run-systemd\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-run-ovn\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-conf-dir\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/773f5b77-c88f-4e79-8104-fd4f9bc8c196-systemd-units\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/18bb7939-b868-42f1-9c76-e12060047d00-iptables-alerter-script\") pod \"iptables-alerter-pj8vm\" (UID: \"18bb7939-b868-42f1-9c76-e12060047d00\") " pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.680808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5aca247b-df86-41d9-9b9d-c9e302f8df71-multus-conf-dir\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.682638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.681158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/da6de3d9-7174-44f8-9773-fc7f5b353bd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.683138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.681252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/773f5b77-c88f-4e79-8104-fd4f9bc8c196-ovnkube-script-lib\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.683453 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.683435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/773f5b77-c88f-4e79-8104-fd4f9bc8c196-ovn-node-metrics-cert\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.690898 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.690877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmp7l\" (UniqueName: \"kubernetes.io/projected/18bb7939-b868-42f1-9c76-e12060047d00-kube-api-access-mmp7l\") pod \"iptables-alerter-pj8vm\" (UID: \"18bb7939-b868-42f1-9c76-e12060047d00\") " pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.691108 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.691096    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:28.691154 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.691111    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:28.691154 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.691121    2563 projected.go:196] Error preparing data for projected volume kube-api-access-rvs4w for pod openshift-network-diagnostics/network-check-target-2jq5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:28.691214 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.691173    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w podName:0e74454c-5e31-4943-881c-02cabbbd5bfc nodeName:}" failed. No retries permitted until 2026-09-09 21:38:29.191160056 +0000 UTC m=+2.087826042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rvs4w" (UniqueName: "kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w") pod "network-check-target-2jq5r" (UID: "0e74454c-5e31-4943-881c-02cabbbd5bfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:28.693864 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.693841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6t8r\" (UniqueName: \"kubernetes.io/projected/5aca247b-df86-41d9-9b9d-c9e302f8df71-kube-api-access-w6t8r\") pod \"multus-krpdt\" (UID: \"5aca247b-df86-41d9-9b9d-c9e302f8df71\") " pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.693969 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.693912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4pp\" (UniqueName: \"kubernetes.io/projected/da6de3d9-7174-44f8-9773-fc7f5b353bd8-kube-api-access-db4pp\") pod \"multus-additional-cni-plugins-wnfn6\" (UID: \"da6de3d9-7174-44f8-9773-fc7f5b353bd8\") " pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.694040 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.693980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6jm\" (UniqueName: \"kubernetes.io/projected/f9a1488a-6bce-4e7b-846d-37825f27b619-kube-api-access-2h6jm\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:28.694154 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.694135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-799n6\" (UniqueName: \"kubernetes.io/projected/773f5b77-c88f-4e79-8104-fd4f9bc8c196-kube-api-access-799n6\") pod \"ovnkube-node-zchqg\" (UID: \"773f5b77-c88f-4e79-8104-fd4f9bc8c196\") " pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.767234 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.767040    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1034d32653d0ee36541005678f807d7f.slice/crio-8f7ad2e6a05830cf65720da2da54b887ada50829a1b20fd2f2f760b0b5d10a3f WatchSource:0}: Error finding container 8f7ad2e6a05830cf65720da2da54b887ada50829a1b20fd2f2f760b0b5d10a3f: Status 404 returned error can't find the container with id 8f7ad2e6a05830cf65720da2da54b887ada50829a1b20fd2f2f760b0b5d10a3f
Sep 09 21:38:28.767566 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.767548    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729b4e6e5ecfd9ee4a0660c00d9444b6.slice/crio-ebc1184ef94d852d058bf2b6fa9925b7f1b5a3b56f12c9d4199fce2544cdd90d WatchSource:0}: Error finding container ebc1184ef94d852d058bf2b6fa9925b7f1b5a3b56f12c9d4199fce2544cdd90d: Status 404 returned error can't find the container with id ebc1184ef94d852d058bf2b6fa9925b7f1b5a3b56f12c9d4199fce2544cdd90d
Sep 09 21:38:28.771594 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.771580    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:38:28.792712 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.792687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:28.797785 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.797756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz"
Sep 09 21:38:28.802483 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.802463    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b64455_a274_4e65_91b6_849f97952b2a.slice/crio-202ba80115b36689e585482ece71ff7e9e336b40ce1dfcc5cf2ff69a16c42772 WatchSource:0}: Error finding container 202ba80115b36689e585482ece71ff7e9e336b40ce1dfcc5cf2ff69a16c42772: Status 404 returned error can't find the container with id 202ba80115b36689e585482ece71ff7e9e336b40ce1dfcc5cf2ff69a16c42772
Sep 09 21:38:28.806313 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.806295    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaafd9b_34bd_42ea_9642_2c99c8204743.slice/crio-db120197c5bafcb39b50b9cf3a41200727c2b7953823c197526b87fbb0f198fa WatchSource:0}: Error finding container db120197c5bafcb39b50b9cf3a41200727c2b7953823c197526b87fbb0f198fa: Status 404 returned error can't find the container with id db120197c5bafcb39b50b9cf3a41200727c2b7953823c197526b87fbb0f198fa
Sep 09 21:38:28.810693 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.810675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kbhzz"
Sep 09 21:38:28.819275 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.819257    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6210d3_1793_4304_9e56_c7c915be4fda.slice/crio-2a11221e5896bed5819f4a8a29944592256f07544c71f9d4bbe769cf034b80de WatchSource:0}: Error finding container 2a11221e5896bed5819f4a8a29944592256f07544c71f9d4bbe769cf034b80de: Status 404 returned error can't find the container with id 2a11221e5896bed5819f4a8a29944592256f07544c71f9d4bbe769cf034b80de
Sep 09 21:38:28.828202 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.828185    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-68g9z"
Sep 09 21:38:28.835789 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.835752    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7198ae4_9d2d_41cb_892c_90e8f729c20e.slice/crio-718db94feb9558badccddcd81f7191f41a929217668a2f8ed5ccdaf00eac4772 WatchSource:0}: Error finding container 718db94feb9558badccddcd81f7191f41a929217668a2f8ed5ccdaf00eac4772: Status 404 returned error can't find the container with id 718db94feb9558badccddcd81f7191f41a929217668a2f8ed5ccdaf00eac4772
Sep 09 21:38:28.843612 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.843597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tlfbl"
Sep 09 21:38:28.850974 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.850956    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715cecb6_2493_4cd4_84b8_6eb84e62fb1b.slice/crio-f414b4e69b5038f5637adfa4d5265cf3cdef4e0e8127bf3278c78616e3a47880 WatchSource:0}: Error finding container f414b4e69b5038f5637adfa4d5265cf3cdef4e0e8127bf3278c78616e3a47880: Status 404 returned error can't find the container with id f414b4e69b5038f5637adfa4d5265cf3cdef4e0e8127bf3278c78616e3a47880
Sep 09 21:38:28.855828 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.855809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wnfn6"
Sep 09 21:38:28.864678 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.864653    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6de3d9_7174_44f8_9773_fc7f5b353bd8.slice/crio-2adc102a5847db2a301d9b2d3dc7bdd84aab6df0b28f0bac277f0c7b25cd625b WatchSource:0}: Error finding container 2adc102a5847db2a301d9b2d3dc7bdd84aab6df0b28f0bac277f0c7b25cd625b: Status 404 returned error can't find the container with id 2adc102a5847db2a301d9b2d3dc7bdd84aab6df0b28f0bac277f0c7b25cd625b
Sep 09 21:38:28.865432 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.865414    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:28.869517 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.869501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-krpdt"
Sep 09 21:38:28.877217 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.877200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aca247b_df86_41d9_9b9d_c9e302f8df71.slice/crio-3ea47c5d3c5bb0ada9e9f8028700119bf54ce48c40f7c84a29fa99b7fbd2b628 WatchSource:0}: Error finding container 3ea47c5d3c5bb0ada9e9f8028700119bf54ce48c40f7c84a29fa99b7fbd2b628: Status 404 returned error can't find the container with id 3ea47c5d3c5bb0ada9e9f8028700119bf54ce48c40f7c84a29fa99b7fbd2b628
Sep 09 21:38:28.898608 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.898591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pj8vm"
Sep 09 21:38:28.905974 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.905956    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bb7939_b868_42f1_9c76_e12060047d00.slice/crio-151d4e8a2e736f1c2bd3ff2188869a08509e9bd1aa74636f6a0c2cdbb6cf4d2b WatchSource:0}: Error finding container 151d4e8a2e736f1c2bd3ff2188869a08509e9bd1aa74636f6a0c2cdbb6cf4d2b: Status 404 returned error can't find the container with id 151d4e8a2e736f1c2bd3ff2188869a08509e9bd1aa74636f6a0c2cdbb6cf4d2b
Sep 09 21:38:28.907271 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:28.907256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:38:28.916258 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:38:28.916238    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773f5b77_c88f_4e79_8104_fd4f9bc8c196.slice/crio-e62ab59de78e56a6e631f8da4defda6aea1ccea6151ba5461a2c82f649f382bc WatchSource:0}: Error finding container e62ab59de78e56a6e631f8da4defda6aea1ccea6151ba5461a2c82f649f382bc: Status 404 returned error can't find the container with id e62ab59de78e56a6e631f8da4defda6aea1ccea6151ba5461a2c82f649f382bc
Sep 09 21:38:28.918591 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.918575    2563 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-zchqg_openshift-ovn-kubernetes(773f5b77-c88f-4e79-8104-fd4f9bc8c196): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 09 21:38:28.919995 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.919977    2563 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-zchqg_openshift-ovn-kubernetes(773f5b77-c88f-4e79-8104-fd4f9bc8c196): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 09 21:38:28.923068 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:28.923021    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" podUID="773f5b77-c88f-4e79-8104-fd4f9bc8c196"
Sep 09 21:38:29.082218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.082191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:29.082350 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:29.082301    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:29.082350 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:29.082344    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret podName:6271f963-afc0-421e-8d4f-cbe5eedbc4d8 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:30.082333088 +0000 UTC m=+2.978999072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret") pod "global-pull-secret-syncer-nd9gr" (UID: "6271f963-afc0-421e-8d4f-cbe5eedbc4d8") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:29.182881 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.182853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:29.183018 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:29.182959    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:29.183018 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:29.183006    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs podName:f9a1488a-6bce-4e7b-846d-37825f27b619 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:30.182990591 +0000 UTC m=+3.079656578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs") pod "network-metrics-daemon-vsdv8" (UID: "f9a1488a-6bce-4e7b-846d-37825f27b619") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:29.283327 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.283254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs4w\" (UniqueName: \"kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w\") pod \"network-check-target-2jq5r\" (UID: \"0e74454c-5e31-4943-881c-02cabbbd5bfc\") " pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:29.283447 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:29.283390    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:29.283447 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:29.283404    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:29.283447 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:29.283411    2563 projected.go:196] Error preparing data for projected volume kube-api-access-rvs4w for pod openshift-network-diagnostics/network-check-target-2jq5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:29.283536 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:29.283457    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w podName:0e74454c-5e31-4943-881c-02cabbbd5bfc nodeName:}" failed. No retries permitted until 2026-09-09 21:38:30.283442631 +0000 UTC m=+3.180108618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvs4w" (UniqueName: "kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w") pod "network-check-target-2jq5r" (UID: "0e74454c-5e31-4943-881c-02cabbbd5bfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:29.519226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.519116    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 21:33:28 +0000 UTC" deadline="2028-05-07 20:37:32.059936999 +0000 UTC"
Sep 09 21:38:29.519226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.519172    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14542h59m2.540769095s"
Sep 09 21:38:29.600793 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.600257    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:29.600793 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:29.600389    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:29.608865 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.608806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" event={"ID":"773f5b77-c88f-4e79-8104-fd4f9bc8c196","Type":"ContainerStarted","Data":"e62ab59de78e56a6e631f8da4defda6aea1ccea6151ba5461a2c82f649f382bc"}
Sep 09 21:38:29.614425 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.614400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-krpdt" event={"ID":"5aca247b-df86-41d9-9b9d-c9e302f8df71","Type":"ContainerStarted","Data":"3ea47c5d3c5bb0ada9e9f8028700119bf54ce48c40f7c84a29fa99b7fbd2b628"}
Sep 09 21:38:29.618026 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.617971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnfn6" event={"ID":"da6de3d9-7174-44f8-9773-fc7f5b353bd8","Type":"ContainerStarted","Data":"2adc102a5847db2a301d9b2d3dc7bdd84aab6df0b28f0bac277f0c7b25cd625b"}
Sep 09 21:38:29.623646 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:29.623565    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" podUID="773f5b77-c88f-4e79-8104-fd4f9bc8c196"
Sep 09 21:38:29.628500 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.628449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tlfbl" event={"ID":"715cecb6-2493-4cd4-84b8-6eb84e62fb1b","Type":"ContainerStarted","Data":"f414b4e69b5038f5637adfa4d5265cf3cdef4e0e8127bf3278c78616e3a47880"}
Sep 09 21:38:29.635366 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.635338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-68g9z" event={"ID":"d7198ae4-9d2d-41cb-892c-90e8f729c20e","Type":"ContainerStarted","Data":"718db94feb9558badccddcd81f7191f41a929217668a2f8ed5ccdaf00eac4772"}
Sep 09 21:38:29.654229 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.654138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5npzg" event={"ID":"c9b64455-a274-4e65-91b6-849f97952b2a","Type":"ContainerStarted","Data":"202ba80115b36689e585482ece71ff7e9e336b40ce1dfcc5cf2ff69a16c42772"}
Sep 09 21:38:29.658121 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.658049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"729b4e6e5ecfd9ee4a0660c00d9444b6","Type":"ContainerStarted","Data":"ebc1184ef94d852d058bf2b6fa9925b7f1b5a3b56f12c9d4199fce2544cdd90d"}
Sep 09 21:38:29.664079 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.664032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" event={"ID":"1034d32653d0ee36541005678f807d7f","Type":"ContainerStarted","Data":"8f7ad2e6a05830cf65720da2da54b887ada50829a1b20fd2f2f760b0b5d10a3f"}
Sep 09 21:38:29.669538 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.669503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pj8vm" event={"ID":"18bb7939-b868-42f1-9c76-e12060047d00","Type":"ContainerStarted","Data":"151d4e8a2e736f1c2bd3ff2188869a08509e9bd1aa74636f6a0c2cdbb6cf4d2b"}
Sep 09 21:38:29.678421 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.678393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kbhzz" event={"ID":"3c6210d3-1793-4304-9e56-c7c915be4fda","Type":"ContainerStarted","Data":"2a11221e5896bed5819f4a8a29944592256f07544c71f9d4bbe769cf034b80de"}
Sep 09 21:38:29.688004 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.687979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz" event={"ID":"6eaafd9b-34bd-42ea-9642-2c99c8204743","Type":"ContainerStarted","Data":"db120197c5bafcb39b50b9cf3a41200727c2b7953823c197526b87fbb0f198fa"}
Sep 09 21:38:29.815215 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:29.815178    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:30.091487 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:30.090759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:30.091487 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:30.091018    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:30.091487 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:30.091100    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret podName:6271f963-afc0-421e-8d4f-cbe5eedbc4d8 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:32.091062116 +0000 UTC m=+4.987728101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret") pod "global-pull-secret-syncer-nd9gr" (UID: "6271f963-afc0-421e-8d4f-cbe5eedbc4d8") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:30.191925 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:30.191387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:30.191925 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:30.191533    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:30.191925 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:30.191587    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs podName:f9a1488a-6bce-4e7b-846d-37825f27b619 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:32.191571053 +0000 UTC m=+5.088237050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs") pod "network-metrics-daemon-vsdv8" (UID: "f9a1488a-6bce-4e7b-846d-37825f27b619") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:30.291796 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:30.291735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs4w\" (UniqueName: \"kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w\") pod \"network-check-target-2jq5r\" (UID: \"0e74454c-5e31-4943-881c-02cabbbd5bfc\") " pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:30.298036 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:30.297937    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:30.298036 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:30.297976    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:30.298036 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:30.297994    2563 projected.go:196] Error preparing data for projected volume kube-api-access-rvs4w for pod openshift-network-diagnostics/network-check-target-2jq5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:30.298267 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:30.298061    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w podName:0e74454c-5e31-4943-881c-02cabbbd5bfc nodeName:}" failed. No retries permitted until 2026-09-09 21:38:32.298035643 +0000 UTC m=+5.194701658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvs4w" (UniqueName: "kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w") pod "network-check-target-2jq5r" (UID: "0e74454c-5e31-4943-881c-02cabbbd5bfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:30.520135 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:30.520032    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 21:33:28 +0000 UTC" deadline="2028-04-09 05:31:41.828303308 +0000 UTC"
Sep 09 21:38:30.520135 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:30.520085    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13855h53m11.308220453s"
Sep 09 21:38:30.600159 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:30.600129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:30.600323 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:30.600242    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:30.600645 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:30.600624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:30.600746 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:30.600724    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:31.599928 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:31.599486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:31.599928 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:31.599605    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:32.106010 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:32.105966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:32.106192 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:32.106128    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:32.106192 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:32.106189    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret podName:6271f963-afc0-421e-8d4f-cbe5eedbc4d8 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:36.10617067 +0000 UTC m=+9.002836673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret") pod "global-pull-secret-syncer-nd9gr" (UID: "6271f963-afc0-421e-8d4f-cbe5eedbc4d8") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:32.206514 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:32.206477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:32.206703 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:32.206609    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:32.206703 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:32.206682    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs podName:f9a1488a-6bce-4e7b-846d-37825f27b619 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:36.206662805 +0000 UTC m=+9.103328806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs") pod "network-metrics-daemon-vsdv8" (UID: "f9a1488a-6bce-4e7b-846d-37825f27b619") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:32.307398 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:32.307364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs4w\" (UniqueName: \"kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w\") pod \"network-check-target-2jq5r\" (UID: \"0e74454c-5e31-4943-881c-02cabbbd5bfc\") " pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:32.307566 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:32.307512    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:32.307566 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:32.307531    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:32.307566 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:32.307544    2563 projected.go:196] Error preparing data for projected volume kube-api-access-rvs4w for pod openshift-network-diagnostics/network-check-target-2jq5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:32.307688 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:32.307611    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w podName:0e74454c-5e31-4943-881c-02cabbbd5bfc nodeName:}" failed. No retries permitted until 2026-09-09 21:38:36.307591815 +0000 UTC m=+9.204257817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvs4w" (UniqueName: "kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w") pod "network-check-target-2jq5r" (UID: "0e74454c-5e31-4943-881c-02cabbbd5bfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:32.599640 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:32.599557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:32.599812 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:32.599688    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:32.599812 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:32.599728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:32.599922 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:32.599823    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:33.600254 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:33.600069    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:33.600254 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:33.600210    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:34.599804 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:34.599436    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:34.599804 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:34.599436    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:34.599804 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:34.599578    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:34.599804 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:34.599668    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:35.599656 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:35.599620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:35.600138 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:35.599806    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:36.137786 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:36.137732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:36.137972 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:36.137906    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:36.138032 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:36.137987    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret podName:6271f963-afc0-421e-8d4f-cbe5eedbc4d8 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:44.137964787 +0000 UTC m=+17.034630775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret") pod "global-pull-secret-syncer-nd9gr" (UID: "6271f963-afc0-421e-8d4f-cbe5eedbc4d8") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:36.238916 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:36.238878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:36.239102 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:36.239053    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:36.239174 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:36.239128    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs podName:f9a1488a-6bce-4e7b-846d-37825f27b619 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:44.239106858 +0000 UTC m=+17.135772865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs") pod "network-metrics-daemon-vsdv8" (UID: "f9a1488a-6bce-4e7b-846d-37825f27b619") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:36.340102 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:36.340067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs4w\" (UniqueName: \"kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w\") pod \"network-check-target-2jq5r\" (UID: \"0e74454c-5e31-4943-881c-02cabbbd5bfc\") " pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:36.340288 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:36.340265    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:36.340350 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:36.340292    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:36.340350 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:36.340306    2563 projected.go:196] Error preparing data for projected volume kube-api-access-rvs4w for pod openshift-network-diagnostics/network-check-target-2jq5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:36.340448 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:36.340368    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w podName:0e74454c-5e31-4943-881c-02cabbbd5bfc nodeName:}" failed. No retries permitted until 2026-09-09 21:38:44.340348348 +0000 UTC m=+17.237014337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvs4w" (UniqueName: "kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w") pod "network-check-target-2jq5r" (UID: "0e74454c-5e31-4943-881c-02cabbbd5bfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:36.600488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:36.600395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:36.600988 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:36.600395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:36.600988 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:36.600542    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:36.600988 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:36.600603    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:37.600205 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:37.600164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:37.600372 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:37.600306    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:38.599825 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.599544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:38.599825 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.599600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:38.600499 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:38.599883    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:38.600499 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:38.599974    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:38.710303 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.710269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5npzg" event={"ID":"c9b64455-a274-4e65-91b6-849f97952b2a","Type":"ContainerStarted","Data":"9d8393a3f6c7ecadc0316923e9a8f4edde79e3a3d4b03888d84f6e02efef2488"}
Sep 09 21:38:38.712088 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.712055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"729b4e6e5ecfd9ee4a0660c00d9444b6","Type":"ContainerStarted","Data":"097375ddf1dc04bd3ab70c73db1878ac8adec85dcaef856af1c6daf9fecfa705"}
Sep 09 21:38:38.713718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.713672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" event={"ID":"1034d32653d0ee36541005678f807d7f","Type":"ContainerStarted","Data":"a4f2d71665d8292ce8f27a2b5013560e7bd73218b9d22af7b7d3267b1c192058"}
Sep 09 21:38:38.715380 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.715356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kbhzz" event={"ID":"3c6210d3-1793-4304-9e56-c7c915be4fda","Type":"ContainerStarted","Data":"94e7acd9a5b70140dbf2b432558ad78d2b41c4d3246329098733de4e05663e3b"}
Sep 09 21:38:38.717110 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.717089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz" event={"ID":"6eaafd9b-34bd-42ea-9642-2c99c8204743","Type":"ContainerStarted","Data":"0574c39c7fa425c8f9cf4e4bf484c87db8584add597e692ecfa55d446cd91a5a"}
Sep 09 21:38:38.718829 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.718806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnfn6" event={"ID":"da6de3d9-7174-44f8-9773-fc7f5b353bd8","Type":"ContainerStarted","Data":"8af0d58afeffcfcfa1cdcc139088731991d910152123092d0ea683277b3863ab"}
Sep 09 21:38:38.720407 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.720384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tlfbl" event={"ID":"715cecb6-2493-4cd4-84b8-6eb84e62fb1b","Type":"ContainerStarted","Data":"49662dbc8e0e2fe5b14901b4e3b1c8984798c9fed25db5a365b7c2dbd1fb15d7"}
Sep 09 21:38:38.722029 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.722007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-68g9z" event={"ID":"d7198ae4-9d2d-41cb-892c-90e8f729c20e","Type":"ContainerStarted","Data":"7a6de7a79d0307ea5ab3ce93cc6fc87923192efb7b71494e661efe0b1ed7ea1b"}
Sep 09 21:38:38.725576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.725537    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5npzg" podStartSLOduration=2.589103256 podStartE2EDuration="11.725522349s" totalImagesPullingTime="9.136419093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:38.725319257 +0000 UTC m=+11.621985265" watchObservedRunningTime="2026-09-09 21:38:38.725522349 +0000 UTC m=+11.622188357"
Sep 09 21:38:38.740557 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.740517    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kbhzz" podStartSLOduration=2.562099994 podStartE2EDuration="11.740501715s" totalImagesPullingTime="9.178401721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:38.740382575 +0000 UTC m=+11.637048584" watchObservedRunningTime="2026-09-09 21:38:38.740501715 +0000 UTC m=+11.637167724"
Sep 09 21:38:38.753382 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.753331    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tlfbl" podStartSLOduration=2.663644167 podStartE2EDuration="11.753310815s" totalImagesPullingTime="9.089666648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:38.752909222 +0000 UTC m=+11.649575253" watchObservedRunningTime="2026-09-09 21:38:38.753310815 +0000 UTC m=+11.649976824"
Sep 09 21:38:38.791849 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.791803    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" podStartSLOduration=10.791786665 podStartE2EDuration="10.791786665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:38.79149397 +0000 UTC m=+11.688159978" watchObservedRunningTime="2026-09-09 21:38:38.791786665 +0000 UTC m=+11.688452681"
Sep 09 21:38:38.809008 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:38.808952    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-68g9z" podStartSLOduration=2.714282297 podStartE2EDuration="11.808935712s" totalImagesPullingTime="9.094653415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:38.808839518 +0000 UTC m=+11.705505554" watchObservedRunningTime="2026-09-09 21:38:38.808935712 +0000 UTC m=+11.705601737"
Sep 09 21:38:39.600355 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:39.600271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:39.600740 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:39.600424    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:39.725737 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:39.725700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pj8vm" event={"ID":"18bb7939-b868-42f1-9c76-e12060047d00","Type":"ContainerStarted","Data":"8c59438257e7e5ef304b2556677f213fccd1c3849cbb84c6199ff1312cbb8d04"}
Sep 09 21:38:39.741727 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:39.741660    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pj8vm" podStartSLOduration=3.680633195 podStartE2EDuration="12.741644425s" totalImagesPullingTime="9.06101123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:39.741153061 +0000 UTC m=+12.637819088" watchObservedRunningTime="2026-09-09 21:38:39.741644425 +0000 UTC m=+12.638310432"
Sep 09 21:38:40.600284 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:40.600250    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:40.600462 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:40.600257    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:40.600462 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:40.600373    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:40.600462 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:40.600430    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:41.599985 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:41.599948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:41.600169 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:41.600106    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:42.600360 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:42.600277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:42.600360 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:42.600302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:42.600882 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:42.600410    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:42.600882 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:42.600549    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:42.732779 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:42.732627    2563 generic.go:356] "Generic (PLEG): container finished" podID="729b4e6e5ecfd9ee4a0660c00d9444b6" containerID="097375ddf1dc04bd3ab70c73db1878ac8adec85dcaef856af1c6daf9fecfa705" exitCode=0
Sep 09 21:38:42.732873 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:42.732695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"729b4e6e5ecfd9ee4a0660c00d9444b6","Type":"ContainerDied","Data":"097375ddf1dc04bd3ab70c73db1878ac8adec85dcaef856af1c6daf9fecfa705"}
Sep 09 21:38:42.734331 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:42.734304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-krpdt" event={"ID":"5aca247b-df86-41d9-9b9d-c9e302f8df71","Type":"ContainerStarted","Data":"e4862697d5124274f46970a85c73981fa37c2de432829a25ad10742b31329fb2"}
Sep 09 21:38:42.735950 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:42.735929    2563 generic.go:356] "Generic (PLEG): container finished" podID="da6de3d9-7174-44f8-9773-fc7f5b353bd8" containerID="8af0d58afeffcfcfa1cdcc139088731991d910152123092d0ea683277b3863ab" exitCode=0
Sep 09 21:38:42.736033 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:42.735987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnfn6" event={"ID":"da6de3d9-7174-44f8-9773-fc7f5b353bd8","Type":"ContainerDied","Data":"8af0d58afeffcfcfa1cdcc139088731991d910152123092d0ea683277b3863ab"}
Sep 09 21:38:42.747471 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:42.747447    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 21:38:43.545600 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:43.545474    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T21:38:42.747468457Z","UUID":"9deaf918-9b6b-46af-bfaa-f04867cc2ff1","Handler":null,"Name":"","Endpoint":""}
Sep 09 21:38:43.547313 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:43.547291    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 21:38:43.547444 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:43.547343    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 21:38:43.567403 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:43.567375    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:43.567945 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:43.567924    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:43.600294 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:43.599968    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:43.600294 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:43.600100    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:43.609232 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:43.609184    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-krpdt" podStartSLOduration=3.326189238 podStartE2EDuration="16.609162362s" totalImagesPullingTime="13.282973124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:42.783316535 +0000 UTC m=+15.679982543" watchObservedRunningTime="2026-09-09 21:38:43.609162362 +0000 UTC m=+16.505828374"
Sep 09 21:38:43.739597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:43.739565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"729b4e6e5ecfd9ee4a0660c00d9444b6","Type":"ContainerStarted","Data":"822a35195401aa6df0011c6e1934ef6758e862ddfc45d9e0a61c332f50026f2f"}
Sep 09 21:38:43.741168 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:43.741142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz" event={"ID":"6eaafd9b-34bd-42ea-9642-2c99c8204743","Type":"ContainerStarted","Data":"d19dd18a420f8fad70d1b850afd8a3bf3f1e0555aab192a5e54be6fc267e7873"}
Sep 09 21:38:43.754056 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:43.754017    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" podStartSLOduration=15.754002492 podStartE2EDuration="15.754002492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:43.753423948 +0000 UTC m=+16.650089955" watchObservedRunningTime="2026-09-09 21:38:43.754002492 +0000 UTC m=+16.650668496"
Sep 09 21:38:44.196160 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:44.196076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:44.196339 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:44.196223    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:44.196405 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:44.196349    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret podName:6271f963-afc0-421e-8d4f-cbe5eedbc4d8 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:00.196290462 +0000 UTC m=+33.092956454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret") pod "global-pull-secret-syncer-nd9gr" (UID: "6271f963-afc0-421e-8d4f-cbe5eedbc4d8") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:44.238105 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:44.238076    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:44.238624 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:44.238600    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5npzg"
Sep 09 21:38:44.296567 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:44.296532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:44.296744 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:44.296677    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:44.296806 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:44.296750    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs podName:f9a1488a-6bce-4e7b-846d-37825f27b619 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:00.296729623 +0000 UTC m=+33.193395622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs") pod "network-metrics-daemon-vsdv8" (UID: "f9a1488a-6bce-4e7b-846d-37825f27b619") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:44.397407 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:44.397373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs4w\" (UniqueName: \"kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w\") pod \"network-check-target-2jq5r\" (UID: \"0e74454c-5e31-4943-881c-02cabbbd5bfc\") " pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:44.397604 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:44.397535    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:44.397604 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:44.397555    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:44.397604 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:44.397567    2563 projected.go:196] Error preparing data for projected volume kube-api-access-rvs4w for pod openshift-network-diagnostics/network-check-target-2jq5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:44.397800 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:44.397616    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w podName:0e74454c-5e31-4943-881c-02cabbbd5bfc nodeName:}" failed. No retries permitted until 2026-09-09 21:39:00.3976017 +0000 UTC m=+33.294267705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvs4w" (UniqueName: "kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w") pod "network-check-target-2jq5r" (UID: "0e74454c-5e31-4943-881c-02cabbbd5bfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:44.600106 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:44.600068    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:44.600287 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:44.600207    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:44.600621 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:44.600599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:44.601127 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:44.600701    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:44.746257 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:44.746224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz" event={"ID":"6eaafd9b-34bd-42ea-9642-2c99c8204743","Type":"ContainerStarted","Data":"5db0f2d9825e222b400e13bdcee214b8c6809fe4c40551333ca4cb6c8b8fa691"}
Sep 09 21:38:45.601309 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:45.601274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:45.601927 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:45.601427    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:46.599738 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:46.599685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:46.599955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:46.599688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:46.599955 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:46.599819    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:46.599955 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:46.599900    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:47.603558 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:47.603517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:47.604030 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:47.603658    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:48.599945 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:48.599921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:48.600070 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:48.599976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:48.600229 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:48.600077    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:48.600229 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:48.600166    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:48.756131 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:48.756061    2563 generic.go:356] "Generic (PLEG): container finished" podID="da6de3d9-7174-44f8-9773-fc7f5b353bd8" containerID="9c8c3642ef91d4fadc04f47087c8e29415a298bede679d0788e8eee031c552ab" exitCode=0
Sep 09 21:38:48.756131 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:48.756104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnfn6" event={"ID":"da6de3d9-7174-44f8-9773-fc7f5b353bd8","Type":"ContainerDied","Data":"9c8c3642ef91d4fadc04f47087c8e29415a298bede679d0788e8eee031c552ab"}
Sep 09 21:38:48.778330 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:48.778278    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzjcz" podStartSLOduration=7.341013373 podStartE2EDuration="21.778262499s" totalImagesPullingTime="14.437249126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:44.763614393 +0000 UTC m=+17.660280401" watchObservedRunningTime="2026-09-09 21:38:48.778262499 +0000 UTC m=+21.674928508"
Sep 09 21:38:49.603289 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:49.603261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:49.603522 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:49.603382    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:50.600443 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:50.600410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:50.600911 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:50.600452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:50.600911 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:50.600532    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:50.600911 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:50.600672    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:50.763091 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:50.763049    2563 generic.go:356] "Generic (PLEG): container finished" podID="da6de3d9-7174-44f8-9773-fc7f5b353bd8" containerID="51e36d59c6e9b304b1ca9dce68229b32975c193b02c65b651234b708337c3a5a" exitCode=0
Sep 09 21:38:50.763308 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:50.763110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnfn6" event={"ID":"da6de3d9-7174-44f8-9773-fc7f5b353bd8","Type":"ContainerDied","Data":"51e36d59c6e9b304b1ca9dce68229b32975c193b02c65b651234b708337c3a5a"}
Sep 09 21:38:51.600232 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:51.600190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:51.600416 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:51.600335    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:52.600491 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:52.600456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:52.600969 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:52.600473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:52.600969 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:52.600572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:52.600969 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:52.600686    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:53.603500 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:53.603247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:53.603932 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:53.603608    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:54.599620 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:54.599594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:54.599827 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:54.599594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:54.599827 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:54.599688    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:54.599827 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:54.599801    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:55.600106 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:55.600079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:55.600491 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:55.600224    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:55.778220 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:55.778184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" event={"ID":"773f5b77-c88f-4e79-8104-fd4f9bc8c196","Type":"ContainerStarted","Data":"45640a2ef7085cf562418e58d9763e43a49a4a61fe08665bedb0b6d240aeb9e3"}
Sep 09 21:38:55.778403 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:55.778274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" event={"ID":"773f5b77-c88f-4e79-8104-fd4f9bc8c196","Type":"ContainerStarted","Data":"506b5f828f8decbcae9500d68339ea0a442fc0c97e4640e0cd89bb3b5a4b3174"}
Sep 09 21:38:55.778403 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:55.778319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" event={"ID":"773f5b77-c88f-4e79-8104-fd4f9bc8c196","Type":"ContainerStarted","Data":"298261a1d33263a7b65830939203330b8c93ebeab3e5f171550d153f6636d635"}
Sep 09 21:38:55.778403 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:55.778365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" event={"ID":"773f5b77-c88f-4e79-8104-fd4f9bc8c196","Type":"ContainerStarted","Data":"219dc67982c4c73873b10e7665e8d17b0ba46b94001a2221b0dd1ae373c90cfa"}
Sep 09 21:38:55.778543 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:55.778408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" event={"ID":"773f5b77-c88f-4e79-8104-fd4f9bc8c196","Type":"ContainerStarted","Data":"83b0ef1c702e88ee5bae28944851a9648696a64e941ac4f6dbf0cfac5b7dd6cb"}
Sep 09 21:38:55.778543 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:55.778454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" event={"ID":"773f5b77-c88f-4e79-8104-fd4f9bc8c196","Type":"ContainerStarted","Data":"15cdc2dec6fca55653a7541b3b2110d0925d94708f4f1808729d0b770bef0e18"}
Sep 09 21:38:55.780614 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:55.780589    2563 generic.go:356] "Generic (PLEG): container finished" podID="da6de3d9-7174-44f8-9773-fc7f5b353bd8" containerID="f03e62cd3a623c1ad9e76c82a82655e557f01c11292b8cc877579c93cfcf2518" exitCode=0
Sep 09 21:38:55.780727 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:55.780624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnfn6" event={"ID":"da6de3d9-7174-44f8-9773-fc7f5b353bd8","Type":"ContainerDied","Data":"f03e62cd3a623c1ad9e76c82a82655e557f01c11292b8cc877579c93cfcf2518"}
Sep 09 21:38:56.600379 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:56.600206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:56.600817 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:56.600206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:56.600817 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:56.600445    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:56.600817 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:56.600545    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:57.601572 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:57.601540    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:57.602014 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:57.601673    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:38:58.599929 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:58.599898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:38:58.600105 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:58.600023    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:38:58.600105 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:58.600023    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:38:58.600105 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:58.600086    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:38:58.792189 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:58.792127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" event={"ID":"773f5b77-c88f-4e79-8104-fd4f9bc8c196","Type":"ContainerStarted","Data":"b0bbd7578e4d39cc66d7cd8ea47a1995b5f75de58cf03fb070cd7ea0d65251d4"}
Sep 09 21:38:59.602926 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:38:59.602899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:38:59.603080 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:38:59.603001    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:39:00.212457 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:00.212370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:39:00.212840 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:00.212523    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:39:00.212840 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:00.212595    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret podName:6271f963-afc0-421e-8d4f-cbe5eedbc4d8 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:32.212577549 +0000 UTC m=+65.109243740 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret") pod "global-pull-secret-syncer-nd9gr" (UID: "6271f963-afc0-421e-8d4f-cbe5eedbc4d8") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:39:00.312925 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:00.312885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:39:00.313122 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:00.313057    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:39:00.313187 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:00.313136    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs podName:f9a1488a-6bce-4e7b-846d-37825f27b619 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:32.313115937 +0000 UTC m=+65.209781950 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs") pod "network-metrics-daemon-vsdv8" (UID: "f9a1488a-6bce-4e7b-846d-37825f27b619") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:39:00.413703 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:00.413659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs4w\" (UniqueName: \"kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w\") pod \"network-check-target-2jq5r\" (UID: \"0e74454c-5e31-4943-881c-02cabbbd5bfc\") " pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:00.413887 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:00.413825    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:39:00.413887 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:00.413849    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:39:00.413887 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:00.413859    2563 projected.go:196] Error preparing data for projected volume kube-api-access-rvs4w for pod openshift-network-diagnostics/network-check-target-2jq5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:39:00.414022 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:00.413918    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w podName:0e74454c-5e31-4943-881c-02cabbbd5bfc nodeName:}" failed. No retries permitted until 2026-09-09 21:39:32.413898067 +0000 UTC m=+65.310564055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvs4w" (UniqueName: "kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w") pod "network-check-target-2jq5r" (UID: "0e74454c-5e31-4943-881c-02cabbbd5bfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:39:00.599915 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:00.599882    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:00.599915 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:00.599907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:39:00.600289 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:00.600012    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:39:00.600289 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:00.600145    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:39:01.603007 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:01.602937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:39:01.603568 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:01.603038    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:39:01.802057 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:01.802024    2563 generic.go:356] "Generic (PLEG): container finished" podID="da6de3d9-7174-44f8-9773-fc7f5b353bd8" containerID="b79e30a092f418932fd3cb63590075988ff357d93815e4d7d25fb75be8adf8ed" exitCode=0
Sep 09 21:39:01.802199 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:01.802092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnfn6" event={"ID":"da6de3d9-7174-44f8-9773-fc7f5b353bd8","Type":"ContainerDied","Data":"b79e30a092f418932fd3cb63590075988ff357d93815e4d7d25fb75be8adf8ed"}
Sep 09 21:39:01.806113 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:01.806077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" event={"ID":"773f5b77-c88f-4e79-8104-fd4f9bc8c196","Type":"ContainerStarted","Data":"f15acdb7b0b5db6944de88cc48d658b57e32f8e19b9b0c4a09d6bbaed815b327"}
Sep 09 21:39:01.806362 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:01.806328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:39:01.806462 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:01.806402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:39:01.824011 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:01.823988    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:39:02.600239 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:02.600206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:39:02.600388 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:02.600311    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:39:02.600388 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:02.600363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:02.600494 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:02.600442    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:39:02.812150 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:02.811970    2563 generic.go:356] "Generic (PLEG): container finished" podID="da6de3d9-7174-44f8-9773-fc7f5b353bd8" containerID="765d896b4eeee8f014785164abca879765c0b57c53c3d63d18b9716896746934" exitCode=0
Sep 09 21:39:02.812492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:02.812051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnfn6" event={"ID":"da6de3d9-7174-44f8-9773-fc7f5b353bd8","Type":"ContainerDied","Data":"765d896b4eeee8f014785164abca879765c0b57c53c3d63d18b9716896746934"}
Sep 09 21:39:02.812600 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:02.812582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:39:02.830464 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:02.830437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:39:02.835881 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:02.835831    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg" podStartSLOduration=9.881490272 podStartE2EDuration="35.835813104s" totalImagesPullingTime="25.954322832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-09-09 21:39:01.869624761 +0000 UTC m=+34.766290782" watchObservedRunningTime="2026-09-09 21:39:02.835813104 +0000 UTC m=+35.732479127"
Sep 09 21:39:03.385761 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:03.385730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nd9gr"]
Sep 09 21:39:03.385937 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:03.385922    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:39:03.386062 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:03.386038    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:39:03.392296 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:03.392272    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vsdv8"]
Sep 09 21:39:03.392425 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:03.392381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:39:03.392492 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:03.392473    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:39:03.401020 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:03.400984    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2jq5r"]
Sep 09 21:39:03.401133 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:03.401076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:03.401205 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:03.401148    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:39:03.817629 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:03.817598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnfn6" event={"ID":"da6de3d9-7174-44f8-9773-fc7f5b353bd8","Type":"ContainerStarted","Data":"1f9c97105cb57bb9949fbc3eecf62ff246dbd875823d3e2434ac8086d6c96a7f"}
Sep 09 21:39:03.870048 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:03.869987    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wnfn6" podStartSLOduration=10.881862145 podStartE2EDuration="36.869972731s" totalImagesPullingTime="25.988110586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:03.869451523 +0000 UTC m=+36.766117531" watchObservedRunningTime="2026-09-09 21:39:03.869972731 +0000 UTC m=+36.766638731"
Sep 09 21:39:05.599445 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:05.599407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:39:05.599912 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:05.599457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:39:05.599912 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:05.599407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:05.599912 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:05.599522    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:39:05.599912 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:05.599600    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:39:05.599912 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:05.599678    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:39:07.600764 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:07.600732    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:07.601178 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:07.600820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:39:07.601178 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:07.600837    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jq5r" podUID="0e74454c-5e31-4943-881c-02cabbbd5bfc"
Sep 09 21:39:07.601178 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:07.600921    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd9gr" podUID="6271f963-afc0-421e-8d4f-cbe5eedbc4d8"
Sep 09 21:39:07.601178 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:07.600961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:39:07.601178 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:07.601057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:39:08.890232 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.890163    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeReady"
Sep 09 21:39:08.890596 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.890265    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 21:39:08.922207 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.922179    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-vs5kp"]
Sep 09 21:39:08.952468 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.952440    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b6d496f4d-s986w"]
Sep 09 21:39:08.952634 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.952567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:08.954798 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.954762    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 21:39:08.954952 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.954934    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 21:39:08.955029 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.955013    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xrjln\""
Sep 09 21:39:08.976460 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.976435    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-vs5kp"]
Sep 09 21:39:08.976460 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.976466    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b6d496f4d-s986w"]
Sep 09 21:39:08.976609 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.976480    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zh6lr"]
Sep 09 21:39:08.976609 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.976551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:08.978958 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.978939    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 21:39:08.979076 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.978970    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 21:39:08.979076 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.978978    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xvgqt\""
Sep 09 21:39:08.979076 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.978973    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 21:39:08.984116 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:08.984096    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 21:39:09.000589 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.000568    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n6cbw"]
Sep 09 21:39:09.000701 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.000687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.003066 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.003045    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8h77q\""
Sep 09 21:39:09.003066 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.003054    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 21:39:09.003066 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.003061    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 21:39:09.024583 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.024559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zh6lr"]
Sep 09 21:39:09.024670 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.024589    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n6cbw"]
Sep 09 21:39:09.024733 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.024692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n6cbw"
Sep 09 21:39:09.027213 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.027189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 21:39:09.027298 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.027286    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bllts\""
Sep 09 21:39:09.027298 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.027290    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 21:39:09.027446 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.027430    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 21:39:09.070949 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.070911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.071075 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.070966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-certificates\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.071075 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.070985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:09.071075 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/888f9b50-262e-4525-a9f2-3857190be163-config-volume\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.071180 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.071180 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-bound-sa-token\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.071180 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhtt\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-kube-api-access-6vhtt\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.071180 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888f9b50-262e-4525-a9f2-3857190be163-tmp-dir\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.071297 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jld\" (UniqueName: \"kubernetes.io/projected/888f9b50-262e-4525-a9f2-3857190be163-kube-api-access-m8jld\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.071297 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-trusted-ca\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.071297 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-installation-pull-secrets\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.071297 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-image-registry-private-configuration\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.071297 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9481e7b-af31-48ec-9504-a706c8d02cbd-ca-trust-extracted\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.071440 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.071308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f742f48a-2f95-4121-adaf-4a1f023498df-nginx-conf\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:09.172291 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.172291 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-certificates\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.172484 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:09.172484 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/888f9b50-262e-4525-a9f2-3857190be163-config-volume\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.172484 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.172484 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-bound-sa-token\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.172484 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.172402    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:39:09.172484 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhtt\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-kube-api-access-6vhtt\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.172484 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888f9b50-262e-4525-a9f2-3857190be163-tmp-dir\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.172484 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jld\" (UniqueName: \"kubernetes.io/projected/888f9b50-262e-4525-a9f2-3857190be163-kube-api-access-m8jld\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.172863 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-trusted-ca\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.172863 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-installation-pull-secrets\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.172863 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.172423    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6d496f4d-s986w: secret "image-registry-tls" not found
Sep 09 21:39:09.172863 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.172709    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls podName:b9481e7b-af31-48ec-9504-a706c8d02cbd nodeName:}" failed. No retries permitted until 2026-09-09 21:39:09.67268648 +0000 UTC m=+42.569352468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls") pod "image-registry-b6d496f4d-s986w" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd") : secret "image-registry-tls" not found
Sep 09 21:39:09.172863 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.172711    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 21:39:09.172863 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.172801    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert podName:f742f48a-2f95-4121-adaf-4a1f023498df nodeName:}" failed. No retries permitted until 2026-09-09 21:39:09.672785071 +0000 UTC m=+42.569451079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-vs5kp" (UID: "f742f48a-2f95-4121-adaf-4a1f023498df") : secret "networking-console-plugin-cert" not found
Sep 09 21:39:09.172863 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.172815    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:09.173218 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.172909    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls podName:888f9b50-262e-4525-a9f2-3857190be163 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:09.672888698 +0000 UTC m=+42.569554698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls") pod "dns-default-zh6lr" (UID: "888f9b50-262e-4525-a9f2-3857190be163") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:09.173218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888f9b50-262e-4525-a9f2-3857190be163-tmp-dir\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.173218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7qx\" (UniqueName: \"kubernetes.io/projected/0b8f9430-2b08-481b-9d13-4290193c0092-kube-api-access-kf7qx\") pod \"ingress-canary-n6cbw\" (UID: \"0b8f9430-2b08-481b-9d13-4290193c0092\") " pod="openshift-ingress-canary/ingress-canary-n6cbw"
Sep 09 21:39:09.173218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.172982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-image-registry-private-configuration\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.173218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.173021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9481e7b-af31-48ec-9504-a706c8d02cbd-ca-trust-extracted\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.173218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.173053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b8f9430-2b08-481b-9d13-4290193c0092-cert\") pod \"ingress-canary-n6cbw\" (UID: \"0b8f9430-2b08-481b-9d13-4290193c0092\") " pod="openshift-ingress-canary/ingress-canary-n6cbw"
Sep 09 21:39:09.173218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.173096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f742f48a-2f95-4121-adaf-4a1f023498df-nginx-conf\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:09.173535 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.173227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/888f9b50-262e-4525-a9f2-3857190be163-config-volume\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.173535 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.173255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-certificates\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.173535 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.173522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9481e7b-af31-48ec-9504-a706c8d02cbd-ca-trust-extracted\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.173822 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.173800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f742f48a-2f95-4121-adaf-4a1f023498df-nginx-conf\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:09.174058 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.174037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-trusted-ca\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.177087 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.177062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-image-registry-private-configuration\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.177087 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.177072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-installation-pull-secrets\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.182120 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.182096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-bound-sa-token\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.182540 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.182520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jld\" (UniqueName: \"kubernetes.io/projected/888f9b50-262e-4525-a9f2-3857190be163-kube-api-access-m8jld\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.182708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.182692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhtt\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-kube-api-access-6vhtt\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.274356 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.274325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b8f9430-2b08-481b-9d13-4290193c0092-cert\") pod \"ingress-canary-n6cbw\" (UID: \"0b8f9430-2b08-481b-9d13-4290193c0092\") " pod="openshift-ingress-canary/ingress-canary-n6cbw"
Sep 09 21:39:09.274517 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.274427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7qx\" (UniqueName: \"kubernetes.io/projected/0b8f9430-2b08-481b-9d13-4290193c0092-kube-api-access-kf7qx\") pod \"ingress-canary-n6cbw\" (UID: \"0b8f9430-2b08-481b-9d13-4290193c0092\") " pod="openshift-ingress-canary/ingress-canary-n6cbw"
Sep 09 21:39:09.276748 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.276717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b8f9430-2b08-481b-9d13-4290193c0092-cert\") pod \"ingress-canary-n6cbw\" (UID: \"0b8f9430-2b08-481b-9d13-4290193c0092\") " pod="openshift-ingress-canary/ingress-canary-n6cbw"
Sep 09 21:39:09.282329 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.282297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7qx\" (UniqueName: \"kubernetes.io/projected/0b8f9430-2b08-481b-9d13-4290193c0092-kube-api-access-kf7qx\") pod \"ingress-canary-n6cbw\" (UID: \"0b8f9430-2b08-481b-9d13-4290193c0092\") " pod="openshift-ingress-canary/ingress-canary-n6cbw"
Sep 09 21:39:09.332099 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.332070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n6cbw"
Sep 09 21:39:09.506879 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.506851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:09.590493 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.590459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:09.600479 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.600453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:39:09.600479 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.600480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:09.600642 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.600464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:39:09.603200 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.603176    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 21:39:09.603365 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.603348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6js7k\""
Sep 09 21:39:09.603522 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.603388    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 21:39:09.603522 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.603450    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 21:39:09.603522 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.603391    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 21:39:09.603522 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.603490    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-znm4z\""
Sep 09 21:39:09.632361 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.632339    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n6cbw"]
Sep 09 21:39:09.635530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:39:09.635510    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8f9430_2b08_481b_9d13_4290193c0092.slice/crio-0e9beafeadb4e4278f92e1bb3c755a345f22d3d10a3bb6effe72ea94fb693498 WatchSource:0}: Error finding container 0e9beafeadb4e4278f92e1bb3c755a345f22d3d10a3bb6effe72ea94fb693498: Status 404 returned error can't find the container with id 0e9beafeadb4e4278f92e1bb3c755a345f22d3d10a3bb6effe72ea94fb693498
Sep 09 21:39:09.676703 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.676671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:09.676843 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.676717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:09.676843 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.676819    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 21:39:09.676843 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.676833    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:39:09.676953 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.676852    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6d496f4d-s986w: secret "image-registry-tls" not found
Sep 09 21:39:09.676953 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.676872    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert podName:f742f48a-2f95-4121-adaf-4a1f023498df nodeName:}" failed. No retries permitted until 2026-09-09 21:39:10.67686039 +0000 UTC m=+43.573526399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-vs5kp" (UID: "f742f48a-2f95-4121-adaf-4a1f023498df") : secret "networking-console-plugin-cert" not found
Sep 09 21:39:09.676953 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.676894    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls podName:b9481e7b-af31-48ec-9504-a706c8d02cbd nodeName:}" failed. No retries permitted until 2026-09-09 21:39:10.676879618 +0000 UTC m=+43.573545624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls") pod "image-registry-b6d496f4d-s986w" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd") : secret "image-registry-tls" not found
Sep 09 21:39:09.676953 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.676896    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:09.677107 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:09.676961    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls podName:888f9b50-262e-4525-a9f2-3857190be163 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:10.676952268 +0000 UTC m=+43.573618258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls") pod "dns-default-zh6lr" (UID: "888f9b50-262e-4525-a9f2-3857190be163") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:09.677107 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.676830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:09.833230 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.833148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n6cbw" event={"ID":"0b8f9430-2b08-481b-9d13-4290193c0092","Type":"ContainerStarted","Data":"0e9beafeadb4e4278f92e1bb3c755a345f22d3d10a3bb6effe72ea94fb693498"}
Sep 09 21:39:09.864374 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.864343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:09.966435 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:09.966394    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:10.688573 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:10.688537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:10.688735 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:10.688588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:10.688735 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:10.688615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:10.688735 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:10.688694    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:39:10.688735 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:10.688715    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6d496f4d-s986w: secret "image-registry-tls" not found
Sep 09 21:39:10.688735 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:10.688718    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:10.688735 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:10.688694    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 21:39:10.688934 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:10.688785    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls podName:b9481e7b-af31-48ec-9504-a706c8d02cbd nodeName:}" failed. No retries permitted until 2026-09-09 21:39:12.688752708 +0000 UTC m=+45.585418695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls") pod "image-registry-b6d496f4d-s986w" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd") : secret "image-registry-tls" not found
Sep 09 21:39:10.688934 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:10.688799    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls podName:888f9b50-262e-4525-a9f2-3857190be163 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:12.688792528 +0000 UTC m=+45.585458514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls") pod "dns-default-zh6lr" (UID: "888f9b50-262e-4525-a9f2-3857190be163") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:10.688934 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:10.688809    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert podName:f742f48a-2f95-4121-adaf-4a1f023498df nodeName:}" failed. No retries permitted until 2026-09-09 21:39:12.688803623 +0000 UTC m=+45.585469609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-vs5kp" (UID: "f742f48a-2f95-4121-adaf-4a1f023498df") : secret "networking-console-plugin-cert" not found
Sep 09 21:39:11.767739 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:11.767696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:11.820337 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:11.820298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:12.207198 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:12.207116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:12.308762 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:12.308727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:12.703598 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:12.703557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:12.703788 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:12.703635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:12.703788 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:12.703669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:12.703788 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:12.703706    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:12.703788 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:12.703755    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 21:39:12.703788 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:12.703760    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:39:12.703997 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:12.703799    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6d496f4d-s986w: secret "image-registry-tls" not found
Sep 09 21:39:12.703997 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:12.703791    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls podName:888f9b50-262e-4525-a9f2-3857190be163 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:16.703753001 +0000 UTC m=+49.600418987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls") pod "dns-default-zh6lr" (UID: "888f9b50-262e-4525-a9f2-3857190be163") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:12.703997 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:12.703857    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert podName:f742f48a-2f95-4121-adaf-4a1f023498df nodeName:}" failed. No retries permitted until 2026-09-09 21:39:16.703834621 +0000 UTC m=+49.600500621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-vs5kp" (UID: "f742f48a-2f95-4121-adaf-4a1f023498df") : secret "networking-console-plugin-cert" not found
Sep 09 21:39:12.703997 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:12.703875    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls podName:b9481e7b-af31-48ec-9504-a706c8d02cbd nodeName:}" failed. No retries permitted until 2026-09-09 21:39:16.703864581 +0000 UTC m=+49.600530567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls") pod "image-registry-b6d496f4d-s986w" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd") : secret "image-registry-tls" not found
Sep 09 21:39:13.843696 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:13.843662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n6cbw" event={"ID":"0b8f9430-2b08-481b-9d13-4290193c0092","Type":"ContainerStarted","Data":"1c97d8857471fe0d1aff1df1933c8667c242b03e5976b22bbe09a32b96470578"}
Sep 09 21:39:13.858488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:13.858438    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n6cbw" podStartSLOduration=2.681780766 podStartE2EDuration="5.858409405s" totalImagesPullingTime="3.176628639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:13.858243472 +0000 UTC m=+46.754909482" watchObservedRunningTime="2026-09-09 21:39:13.858409405 +0000 UTC m=+46.755075415"
Sep 09 21:39:16.738299 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:16.738258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:16.738661 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:16.738313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:16.738661 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:16.738353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:16.738661 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:16.738430    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:39:16.738661 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:16.738448    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6d496f4d-s986w: secret "image-registry-tls" not found
Sep 09 21:39:16.738661 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:16.738471    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:16.738661 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:16.738474    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 21:39:16.738661 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:16.738505    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls podName:b9481e7b-af31-48ec-9504-a706c8d02cbd nodeName:}" failed. No retries permitted until 2026-09-09 21:39:24.738489298 +0000 UTC m=+57.635155284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls") pod "image-registry-b6d496f4d-s986w" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd") : secret "image-registry-tls" not found
Sep 09 21:39:16.738661 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:16.738530    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert podName:f742f48a-2f95-4121-adaf-4a1f023498df nodeName:}" failed. No retries permitted until 2026-09-09 21:39:24.738512159 +0000 UTC m=+57.635178145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-vs5kp" (UID: "f742f48a-2f95-4121-adaf-4a1f023498df") : secret "networking-console-plugin-cert" not found
Sep 09 21:39:16.738661 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:16.738545    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls podName:888f9b50-262e-4525-a9f2-3857190be163 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:24.738538889 +0000 UTC m=+57.635204875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls") pod "dns-default-zh6lr" (UID: "888f9b50-262e-4525-a9f2-3857190be163") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:19.497446 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:19.497412    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:19.563192 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:19.563154    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:20.356666 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:20.356564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:20.458681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:20.458642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:23.338869 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:23.338825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:23.441073 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:23.441039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:23.540047 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:23.540005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:23.592895 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:23.592814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:24.802581 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:24.802545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:24.803105 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:24.802612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:24.803105 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:24.802667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:24.803105 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:24.802733    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:39:24.803105 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:24.802758    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 21:39:24.803105 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:24.802760    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6d496f4d-s986w: secret "image-registry-tls" not found
Sep 09 21:39:24.803105 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:24.802853    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert podName:f742f48a-2f95-4121-adaf-4a1f023498df nodeName:}" failed. No retries permitted until 2026-09-09 21:39:40.802831535 +0000 UTC m=+73.699497539 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-vs5kp" (UID: "f742f48a-2f95-4121-adaf-4a1f023498df") : secret "networking-console-plugin-cert" not found
Sep 09 21:39:24.803105 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:24.802872    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls podName:b9481e7b-af31-48ec-9504-a706c8d02cbd nodeName:}" failed. No retries permitted until 2026-09-09 21:39:40.802862395 +0000 UTC m=+73.699528382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls") pod "image-registry-b6d496f4d-s986w" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd") : secret "image-registry-tls" not found
Sep 09 21:39:24.803105 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:24.802901    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:24.803105 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:24.802970    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls podName:888f9b50-262e-4525-a9f2-3857190be163 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:40.80295571 +0000 UTC m=+73.699621696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls") pod "dns-default-zh6lr" (UID: "888f9b50-262e-4525-a9f2-3857190be163") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:24.874447 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:24.874415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:24.976391 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:24.976342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:25.032276 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:25.032240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:32.254850 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.254792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:39:32.257200 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.257182    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 21:39:32.267737 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.267701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6271f963-afc0-421e-8d4f-cbe5eedbc4d8-original-pull-secret\") pod \"global-pull-secret-syncer-nd9gr\" (UID: \"6271f963-afc0-421e-8d4f-cbe5eedbc4d8\") " pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:39:32.355982 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.355933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:39:32.358693 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.358677    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 21:39:32.367098 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:32.367074    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 21:39:32.367150 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:32.367136    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs podName:f9a1488a-6bce-4e7b-846d-37825f27b619 nodeName:}" failed. No retries permitted until 2026-09-09 21:40:36.36711972 +0000 UTC m=+129.263785722 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs") pod "network-metrics-daemon-vsdv8" (UID: "f9a1488a-6bce-4e7b-846d-37825f27b619") : secret "metrics-daemon-secret" not found
Sep 09 21:39:32.411477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.411450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd9gr"
Sep 09 21:39:32.456914 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.456882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs4w\" (UniqueName: \"kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w\") pod \"network-check-target-2jq5r\" (UID: \"0e74454c-5e31-4943-881c-02cabbbd5bfc\") " pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:32.459993 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.459951    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 21:39:32.470381 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.470358    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 21:39:32.481542 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.481511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvs4w\" (UniqueName: \"kubernetes.io/projected/0e74454c-5e31-4943-881c-02cabbbd5bfc-kube-api-access-rvs4w\") pod \"network-check-target-2jq5r\" (UID: \"0e74454c-5e31-4943-881c-02cabbbd5bfc\") " pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:32.686126 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.686056    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nd9gr"]
Sep 09 21:39:32.689679 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:39:32.689649    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6271f963_afc0_421e_8d4f_cbe5eedbc4d8.slice/crio-99a048d39d41c7ab9b7e3d4c2194be8760708c581be41489823801c33d12c6d3 WatchSource:0}: Error finding container 99a048d39d41c7ab9b7e3d4c2194be8760708c581be41489823801c33d12c6d3: Status 404 returned error can't find the container with id 99a048d39d41c7ab9b7e3d4c2194be8760708c581be41489823801c33d12c6d3
Sep 09 21:39:32.725354 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.725331    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6js7k\""
Sep 09 21:39:32.733422 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.733404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:32.887161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:32.887131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nd9gr" event={"ID":"6271f963-afc0-421e-8d4f-cbe5eedbc4d8","Type":"ContainerStarted","Data":"99a048d39d41c7ab9b7e3d4c2194be8760708c581be41489823801c33d12c6d3"}
Sep 09 21:39:33.038456 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:33.038426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2jq5r"]
Sep 09 21:39:33.041295 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:39:33.041274    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e74454c_5e31_4943_881c_02cabbbd5bfc.slice/crio-88bb15e7e84f292c9f2d501a05c7df6e2f8983341eec5eedf29251c08c2526d0 WatchSource:0}: Error finding container 88bb15e7e84f292c9f2d501a05c7df6e2f8983341eec5eedf29251c08c2526d0: Status 404 returned error can't find the container with id 88bb15e7e84f292c9f2d501a05c7df6e2f8983341eec5eedf29251c08c2526d0
Sep 09 21:39:33.891144 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:33.891104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2jq5r" event={"ID":"0e74454c-5e31-4943-881c-02cabbbd5bfc","Type":"ContainerStarted","Data":"88bb15e7e84f292c9f2d501a05c7df6e2f8983341eec5eedf29251c08c2526d0"}
Sep 09 21:39:34.833103 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:34.833076    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zchqg"
Sep 09 21:39:34.989440 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:34.989406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:35.044694 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:35.044662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:38.906232 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:38.906179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2jq5r" event={"ID":"0e74454c-5e31-4943-881c-02cabbbd5bfc","Type":"ContainerStarted","Data":"bb5bf4df2a5565d46b8473613c694c82104ef1e4388068d64c9df6d2dbe13d77"}
Sep 09 21:39:38.906714 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:38.906330    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:39:38.907523 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:38.907499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nd9gr" event={"ID":"6271f963-afc0-421e-8d4f-cbe5eedbc4d8","Type":"ContainerStarted","Data":"7fd1a8e2f16b47f444e6f90e6d7ce7060334ee04ea2c4bae34d137387cef5286"}
Sep 09 21:39:38.925151 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:38.925104    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2jq5r" podStartSLOduration=66.515324177 podStartE2EDuration="1m11.92509221s" totalImagesPullingTime="5.409768033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:38.923260157 +0000 UTC m=+71.819926165" watchObservedRunningTime="2026-09-09 21:39:38.92509221 +0000 UTC m=+71.821758215"
Sep 09 21:39:38.938122 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:38.938080    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nd9gr" podStartSLOduration=66.17264329 podStartE2EDuration="1m11.938069279s" totalImagesPullingTime="5.765425989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:38.937954542 +0000 UTC m=+71.834620549" watchObservedRunningTime="2026-09-09 21:39:38.938069279 +0000 UTC m=+71.834735287"
Sep 09 21:39:40.815156 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:40.815114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:39:40.815156 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:40.815159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:39:40.815552 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:40.815207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:39:40.815552 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:40.815265    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:39:40.815552 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:40.815282    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6d496f4d-s986w: secret "image-registry-tls" not found
Sep 09 21:39:40.815552 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:40.815298    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:40.815552 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:40.815336    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 21:39:40.815552 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:40.815343    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls podName:888f9b50-262e-4525-a9f2-3857190be163 nodeName:}" failed. No retries permitted until 2026-09-09 21:40:12.815326522 +0000 UTC m=+105.711992523 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls") pod "dns-default-zh6lr" (UID: "888f9b50-262e-4525-a9f2-3857190be163") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:40.815552 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:40.815407    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls podName:b9481e7b-af31-48ec-9504-a706c8d02cbd nodeName:}" failed. No retries permitted until 2026-09-09 21:40:12.815389484 +0000 UTC m=+105.712055472 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls") pod "image-registry-b6d496f4d-s986w" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd") : secret "image-registry-tls" not found
Sep 09 21:39:40.815552 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:39:40.815418    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert podName:f742f48a-2f95-4121-adaf-4a1f023498df nodeName:}" failed. No retries permitted until 2026-09-09 21:40:12.815412573 +0000 UTC m=+105.712078558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-vs5kp" (UID: "f742f48a-2f95-4121-adaf-4a1f023498df") : secret "networking-console-plugin-cert" not found
Sep 09 21:39:45.330974 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:45.330935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:45.390158 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:45.390099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:46.350241 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:46.350209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:46.409000 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:46.408929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:46.462415 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:39:46.462382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:09.912542 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:40:09.912403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2jq5r"
Sep 09 21:40:12.838272 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:40:12.838235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:40:12.838620 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:40:12.838281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:40:12.838620 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:40:12.838391    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:40:12.838620 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:40:12.838408    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6d496f4d-s986w: secret "image-registry-tls" not found
Sep 09 21:40:12.838620 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:40:12.838428    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 21:40:12.838620 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:40:12.838460    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls podName:b9481e7b-af31-48ec-9504-a706c8d02cbd nodeName:}" failed. No retries permitted until 2026-09-09 21:41:16.838440716 +0000 UTC m=+169.735106709 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls") pod "image-registry-b6d496f4d-s986w" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd") : secret "image-registry-tls" not found
Sep 09 21:40:12.838620 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:40:12.838476    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert podName:f742f48a-2f95-4121-adaf-4a1f023498df nodeName:}" failed. No retries permitted until 2026-09-09 21:41:16.838467717 +0000 UTC m=+169.735133703 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-vs5kp" (UID: "f742f48a-2f95-4121-adaf-4a1f023498df") : secret "networking-console-plugin-cert" not found
Sep 09 21:40:12.838620 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:40:12.838501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:40:12.838620 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:40:12.838597    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:40:12.838891 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:40:12.838663    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls podName:888f9b50-262e-4525-a9f2-3857190be163 nodeName:}" failed. No retries permitted until 2026-09-09 21:41:16.838644222 +0000 UTC m=+169.735310210 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls") pod "dns-default-zh6lr" (UID: "888f9b50-262e-4525-a9f2-3857190be163") : secret "dns-default-metrics-tls" not found
Sep 09 21:40:36.400612 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:40:36.400575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:40:36.401022 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:40:36.400741    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 21:40:36.401022 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:40:36.400826    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs podName:f9a1488a-6bce-4e7b-846d-37825f27b619 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:38.400808283 +0000 UTC m=+251.297474289 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs") pod "network-metrics-daemon-vsdv8" (UID: "f9a1488a-6bce-4e7b-846d-37825f27b619") : secret "metrics-daemon-secret" not found
Sep 09 21:41:11.962961 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:11.962916    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp" podUID="f742f48a-2f95-4121-adaf-4a1f023498df"
Sep 09 21:41:11.986151 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:11.986116    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-b6d496f4d-s986w" podUID="b9481e7b-af31-48ec-9504-a706c8d02cbd"
Sep 09 21:41:12.009272 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:12.009231    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zh6lr" podUID="888f9b50-262e-4525-a9f2-3857190be163"
Sep 09 21:41:12.122128 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:41:12.122098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:41:12.122259 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:41:12.122099    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:41:12.617762 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:12.617708    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:41:16.879189 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:41:16.879134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:41:16.879189 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:41:16.879192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:41:16.879607 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:41:16.879223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:41:16.879607 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:16.879299    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:41:16.879607 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:16.879300    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:41:16.879607 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:16.879322    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6d496f4d-s986w: secret "image-registry-tls" not found
Sep 09 21:41:16.879607 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:16.879352    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 21:41:16.879607 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:16.879359    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls podName:888f9b50-262e-4525-a9f2-3857190be163 nodeName:}" failed. No retries permitted until 2026-09-09 21:43:18.879346604 +0000 UTC m=+291.776012590 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls") pod "dns-default-zh6lr" (UID: "888f9b50-262e-4525-a9f2-3857190be163") : secret "dns-default-metrics-tls" not found
Sep 09 21:41:16.879607 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:16.879417    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls podName:b9481e7b-af31-48ec-9504-a706c8d02cbd nodeName:}" failed. No retries permitted until 2026-09-09 21:43:18.879403969 +0000 UTC m=+291.776069955 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls") pod "image-registry-b6d496f4d-s986w" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd") : secret "image-registry-tls" not found
Sep 09 21:41:16.879607 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:41:16.879428    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert podName:f742f48a-2f95-4121-adaf-4a1f023498df nodeName:}" failed. No retries permitted until 2026-09-09 21:43:18.879421961 +0000 UTC m=+291.776087947 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-vs5kp" (UID: "f742f48a-2f95-4121-adaf-4a1f023498df") : secret "networking-console-plugin-cert" not found
Sep 09 21:41:24.599516 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:41:24.599411    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:41:26.600078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:41:26.600026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:42:10.381796 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:10.381745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:35.077843 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.077809    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-z2v2b"]
Sep 09 21:42:35.079895 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.079869    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"]
Sep 09 21:42:35.080040 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.079985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.081602 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.081581    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"]
Sep 09 21:42:35.081725 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.081699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:35.082591 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.082571    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 21:42:35.082685 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.082666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pfkk9\""
Sep 09 21:42:35.083464 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.083448    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 21:42:35.083546 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.083524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:35.083605 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.083573    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 21:42:35.084514 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.084492    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 21:42:35.084664 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.084648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rg92g\""
Sep 09 21:42:35.086260 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.086241    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 21:42:35.086260 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.086256    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 21:42:35.086523 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.086498    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 21:42:35.087753 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.087735    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 21:42:35.087869 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.087830    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:42:35.087869 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.087831    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 21:42:35.087976 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.087886    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sggdj\""
Sep 09 21:42:35.087976 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.087836    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:42:35.095996 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.093316    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"]
Sep 09 21:42:35.095996 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.094070    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 21:42:35.095996 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.095438    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-z2v2b"]
Sep 09 21:42:35.097947 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.097922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"]
Sep 09 21:42:35.176487 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.176457    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj"]
Sep 09 21:42:35.178220 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.178205    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"]
Sep 09 21:42:35.178344 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.178329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj"
Sep 09 21:42:35.179921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.179903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.181046 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.181024    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 21:42:35.181123 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.181052    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9dz8r\""
Sep 09 21:42:35.181337 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.181321    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:42:35.183193 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.183177    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 21:42:35.183466 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.183446    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:42:35.183559 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.183464    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 21:42:35.184485 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.184470    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 21:42:35.184932 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.184799    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dprk6\""
Sep 09 21:42:35.194563 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.194545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"]
Sep 09 21:42:35.195560 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.195544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj"]
Sep 09 21:42:35.229387 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.229355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/448a39a8-aab5-4696-b385-2169e6ee4a6e-config\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.229521 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.229410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/448a39a8-aab5-4696-b385-2169e6ee4a6e-serving-cert\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.229521 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.229437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5bj\" (UniqueName: \"kubernetes.io/projected/448a39a8-aab5-4696-b385-2169e6ee4a6e-kube-api-access-hq5bj\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.229521 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.229463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/448a39a8-aab5-4696-b385-2169e6ee4a6e-trusted-ca\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.229521 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.229479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5pj8\" (UniqueName: \"kubernetes.io/projected/6ab3349e-53f6-462b-a888-bff0d463fd2f-kube-api-access-z5pj8\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:35.229521 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.229497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ab3349e-53f6-462b-a888-bff0d463fd2f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:35.229521 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.229521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:35.229715 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.229538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vz5q\" (UniqueName: \"kubernetes.io/projected/99f5bc95-24e1-4520-b21f-57edc4133ff8-kube-api-access-9vz5q\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:35.229715 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.229637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:35.312273 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.312243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:35.330760 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.330694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ab3349e-53f6-462b-a888-bff0d463fd2f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:35.330760 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.330741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vz5q\" (UniqueName: \"kubernetes.io/projected/99f5bc95-24e1-4520-b21f-57edc4133ff8-kube-api-access-9vz5q\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:35.330955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.330811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffe13b21-9884-46df-bcdc-169112e4b10d-config\") pod \"service-ca-operator-859f94c4b-9gx7x\" (UID: \"ffe13b21-9884-46df-bcdc-169112e4b10d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.330955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.330845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bn4\" (UniqueName: \"kubernetes.io/projected/ffe13b21-9884-46df-bcdc-169112e4b10d-kube-api-access-k8bn4\") pod \"service-ca-operator-859f94c4b-9gx7x\" (UID: \"ffe13b21-9884-46df-bcdc-169112e4b10d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.330955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.330898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:35.330955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.330947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ffe13b21-9884-46df-bcdc-169112e4b10d-serving-cert\") pod \"service-ca-operator-859f94c4b-9gx7x\" (UID: \"ffe13b21-9884-46df-bcdc-169112e4b10d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.331157 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.330979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/448a39a8-aab5-4696-b385-2169e6ee4a6e-config\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.331157 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.331021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/448a39a8-aab5-4696-b385-2169e6ee4a6e-serving-cert\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.331157 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.331051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5bj\" (UniqueName: \"kubernetes.io/projected/448a39a8-aab5-4696-b385-2169e6ee4a6e-kube-api-access-hq5bj\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.331157 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:35.331075    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:35.331157 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.331097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5pj8\" (UniqueName: \"kubernetes.io/projected/6ab3349e-53f6-462b-a888-bff0d463fd2f-kube-api-access-z5pj8\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:35.331157 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.331131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:35.331430 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:35.331170    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls podName:6ab3349e-53f6-462b-a888-bff0d463fd2f nodeName:}" failed. No retries permitted until 2026-09-09 21:42:35.831150247 +0000 UTC m=+248.727816257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8pcxl" (UID: "6ab3349e-53f6-462b-a888-bff0d463fd2f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:35.331430 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.331208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/448a39a8-aab5-4696-b385-2169e6ee4a6e-trusted-ca\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.331430 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.331242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfkh\" (UniqueName: \"kubernetes.io/projected/b903bce2-59e5-4b6f-9cb5-ebea1800e44f-kube-api-access-zxfkh\") pod \"volume-data-source-validator-695c895bd6-gmnwj\" (UID: \"b903bce2-59e5-4b6f-9cb5-ebea1800e44f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj"
Sep 09 21:42:35.331430 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:35.331401    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 21:42:35.331606 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:35.331486    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls podName:99f5bc95-24e1-4520-b21f-57edc4133ff8 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:35.831468564 +0000 UTC m=+248.728134553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hdssz" (UID: "99f5bc95-24e1-4520-b21f-57edc4133ff8") : secret "samples-operator-tls" not found
Sep 09 21:42:35.331606 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.331489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ab3349e-53f6-462b-a888-bff0d463fd2f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:35.331697 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.331680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/448a39a8-aab5-4696-b385-2169e6ee4a6e-config\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.331998 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.331982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/448a39a8-aab5-4696-b385-2169e6ee4a6e-trusted-ca\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.333527 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.333511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/448a39a8-aab5-4696-b385-2169e6ee4a6e-serving-cert\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.339372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.339351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5pj8\" (UniqueName: \"kubernetes.io/projected/6ab3349e-53f6-462b-a888-bff0d463fd2f-kube-api-access-z5pj8\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:35.340176 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.340149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5bj\" (UniqueName: \"kubernetes.io/projected/448a39a8-aab5-4696-b385-2169e6ee4a6e-kube-api-access-hq5bj\") pod \"console-operator-85b454874b-z2v2b\" (UID: \"448a39a8-aab5-4696-b385-2169e6ee4a6e\") " pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.340374 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.340357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vz5q\" (UniqueName: \"kubernetes.io/projected/99f5bc95-24e1-4520-b21f-57edc4133ff8-kube-api-access-9vz5q\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:35.391299 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.391274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:35.432183 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.432151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffe13b21-9884-46df-bcdc-169112e4b10d-config\") pod \"service-ca-operator-859f94c4b-9gx7x\" (UID: \"ffe13b21-9884-46df-bcdc-169112e4b10d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.432183 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.432187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bn4\" (UniqueName: \"kubernetes.io/projected/ffe13b21-9884-46df-bcdc-169112e4b10d-kube-api-access-k8bn4\") pod \"service-ca-operator-859f94c4b-9gx7x\" (UID: \"ffe13b21-9884-46df-bcdc-169112e4b10d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.432425 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.432214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ffe13b21-9884-46df-bcdc-169112e4b10d-serving-cert\") pod \"service-ca-operator-859f94c4b-9gx7x\" (UID: \"ffe13b21-9884-46df-bcdc-169112e4b10d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.432425 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.432259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfkh\" (UniqueName: \"kubernetes.io/projected/b903bce2-59e5-4b6f-9cb5-ebea1800e44f-kube-api-access-zxfkh\") pod \"volume-data-source-validator-695c895bd6-gmnwj\" (UID: \"b903bce2-59e5-4b6f-9cb5-ebea1800e44f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj"
Sep 09 21:42:35.432812 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.432789    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffe13b21-9884-46df-bcdc-169112e4b10d-config\") pod \"service-ca-operator-859f94c4b-9gx7x\" (UID: \"ffe13b21-9884-46df-bcdc-169112e4b10d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.435378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.434985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ffe13b21-9884-46df-bcdc-169112e4b10d-serving-cert\") pod \"service-ca-operator-859f94c4b-9gx7x\" (UID: \"ffe13b21-9884-46df-bcdc-169112e4b10d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.440469 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.440446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bn4\" (UniqueName: \"kubernetes.io/projected/ffe13b21-9884-46df-bcdc-169112e4b10d-kube-api-access-k8bn4\") pod \"service-ca-operator-859f94c4b-9gx7x\" (UID: \"ffe13b21-9884-46df-bcdc-169112e4b10d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.440961 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.440916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfkh\" (UniqueName: \"kubernetes.io/projected/b903bce2-59e5-4b6f-9cb5-ebea1800e44f-kube-api-access-zxfkh\") pod \"volume-data-source-validator-695c895bd6-gmnwj\" (UID: \"b903bce2-59e5-4b6f-9cb5-ebea1800e44f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj"
Sep 09 21:42:35.487801 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.487747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj"
Sep 09 21:42:35.492187 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.492169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"
Sep 09 21:42:35.656874 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.656791    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-z2v2b"]
Sep 09 21:42:35.660090 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:42:35.660052    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448a39a8_aab5_4696_b385_2169e6ee4a6e.slice/crio-fd38932bd14977cc402ce6bae638df53ec18d315d0a9af6cad77463521d2831d WatchSource:0}: Error finding container fd38932bd14977cc402ce6bae638df53ec18d315d0a9af6cad77463521d2831d: Status 404 returned error can't find the container with id fd38932bd14977cc402ce6bae638df53ec18d315d0a9af6cad77463521d2831d
Sep 09 21:42:35.753885 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.753854    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj"]
Sep 09 21:42:35.756915 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:42:35.756892    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb903bce2_59e5_4b6f_9cb5_ebea1800e44f.slice/crio-5e2f6e0c75abdf7cb4d1122b80492095a3c33778ef71fde29e5c33465d39c86c WatchSource:0}: Error finding container 5e2f6e0c75abdf7cb4d1122b80492095a3c33778ef71fde29e5c33465d39c86c: Status 404 returned error can't find the container with id 5e2f6e0c75abdf7cb4d1122b80492095a3c33778ef71fde29e5c33465d39c86c
Sep 09 21:42:35.759049 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.759015    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x"]
Sep 09 21:42:35.761533 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:42:35.761511    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe13b21_9884_46df_bcdc_169112e4b10d.slice/crio-e75c3dd698c34eed32a2e17ba7c24bf42228755f3950f7cf1534fe2ca6c9afaa WatchSource:0}: Error finding container e75c3dd698c34eed32a2e17ba7c24bf42228755f3950f7cf1534fe2ca6c9afaa: Status 404 returned error can't find the container with id e75c3dd698c34eed32a2e17ba7c24bf42228755f3950f7cf1534fe2ca6c9afaa
Sep 09 21:42:35.835587 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.835548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:35.835791 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:35.835601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:35.835791 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:35.835689    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 21:42:35.835791 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:35.835691    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:35.835791 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:35.835735    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls podName:99f5bc95-24e1-4520-b21f-57edc4133ff8 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:36.835723697 +0000 UTC m=+249.732389683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hdssz" (UID: "99f5bc95-24e1-4520-b21f-57edc4133ff8") : secret "samples-operator-tls" not found
Sep 09 21:42:35.835791 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:35.835754    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls podName:6ab3349e-53f6-462b-a888-bff0d463fd2f nodeName:}" failed. No retries permitted until 2026-09-09 21:42:36.83574136 +0000 UTC m=+249.732407347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8pcxl" (UID: "6ab3349e-53f6-462b-a888-bff0d463fd2f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:36.323135 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:36.323096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z2v2b" event={"ID":"448a39a8-aab5-4696-b385-2169e6ee4a6e","Type":"ContainerStarted","Data":"fd38932bd14977cc402ce6bae638df53ec18d315d0a9af6cad77463521d2831d"}
Sep 09 21:42:36.324495 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:36.324456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x" event={"ID":"ffe13b21-9884-46df-bcdc-169112e4b10d","Type":"ContainerStarted","Data":"e75c3dd698c34eed32a2e17ba7c24bf42228755f3950f7cf1534fe2ca6c9afaa"}
Sep 09 21:42:36.325832 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:36.325808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj" event={"ID":"b903bce2-59e5-4b6f-9cb5-ebea1800e44f","Type":"ContainerStarted","Data":"5e2f6e0c75abdf7cb4d1122b80492095a3c33778ef71fde29e5c33465d39c86c"}
Sep 09 21:42:36.804515 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:36.804481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:36.844866 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:36.844633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:36.844866 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:36.844719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:36.844866 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:36.844832    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:36.844866 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:36.844870    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 21:42:36.845190 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:36.844906    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls podName:6ab3349e-53f6-462b-a888-bff0d463fd2f nodeName:}" failed. No retries permitted until 2026-09-09 21:42:38.84488706 +0000 UTC m=+251.741553062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8pcxl" (UID: "6ab3349e-53f6-462b-a888-bff0d463fd2f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:36.845190 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:36.844928    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls podName:99f5bc95-24e1-4520-b21f-57edc4133ff8 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:38.844917127 +0000 UTC m=+251.741583116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hdssz" (UID: "99f5bc95-24e1-4520-b21f-57edc4133ff8") : secret "samples-operator-tls" not found
Sep 09 21:42:37.331439 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:37.331401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj" event={"ID":"b903bce2-59e5-4b6f-9cb5-ebea1800e44f","Type":"ContainerStarted","Data":"c290ec013431ebd2bc75954f6859694cbf8d0727b84a9093a920823eed13d25a"}
Sep 09 21:42:37.347148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:37.347092    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gmnwj" podStartSLOduration=0.922019951 podStartE2EDuration="2.347074779s" totalImagesPullingTime="1.425054828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:37.346062328 +0000 UTC m=+250.242728338" watchObservedRunningTime="2026-09-09 21:42:37.347074779 +0000 UTC m=+250.243740787"
Sep 09 21:42:37.412377 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:37.412326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:37.514739 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:37.514696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:38.461472 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:38.461444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:42:38.461739 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:38.461598    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 21:42:38.461739 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:38.461655    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs podName:f9a1488a-6bce-4e7b-846d-37825f27b619 nodeName:}" failed. No retries permitted until 2026-09-09 21:44:40.461641164 +0000 UTC m=+373.358307150 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs") pod "network-metrics-daemon-vsdv8" (UID: "f9a1488a-6bce-4e7b-846d-37825f27b619") : secret "metrics-daemon-secret" not found
Sep 09 21:42:38.864736 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:38.864695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:38.864928 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:38.864886    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 21:42:38.864968 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:38.864948    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls podName:99f5bc95-24e1-4520-b21f-57edc4133ff8 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:42.864932367 +0000 UTC m=+255.761598378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hdssz" (UID: "99f5bc95-24e1-4520-b21f-57edc4133ff8") : secret "samples-operator-tls" not found
Sep 09 21:42:38.864968 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:38.864954    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:38.865039 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:38.864995    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls podName:6ab3349e-53f6-462b-a888-bff0d463fd2f nodeName:}" failed. No retries permitted until 2026-09-09 21:42:42.864984246 +0000 UTC m=+255.761650232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8pcxl" (UID: "6ab3349e-53f6-462b-a888-bff0d463fd2f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:38.865039 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:38.864889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:39.339616 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.339571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x" event={"ID":"ffe13b21-9884-46df-bcdc-169112e4b10d","Type":"ContainerStarted","Data":"6bbc05e6a00632b85f74c4e384361858e8b04cf49e3b2ea88b029699123c30b5"}
Sep 09 21:42:39.341180 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.341160    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/0.log"
Sep 09 21:42:39.341271 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.341194    2563 generic.go:356] "Generic (PLEG): container finished" podID="448a39a8-aab5-4696-b385-2169e6ee4a6e" containerID="cd55c3a46d022c9b4a0376516b19cce68af8eab1952136faa1dfd28f585923a8" exitCode=255
Sep 09 21:42:39.341271 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.341228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z2v2b" event={"ID":"448a39a8-aab5-4696-b385-2169e6ee4a6e","Type":"ContainerDied","Data":"cd55c3a46d022c9b4a0376516b19cce68af8eab1952136faa1dfd28f585923a8"}
Sep 09 21:42:39.341432 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.341417    2563 scope.go:117] "RemoveContainer" containerID="cd55c3a46d022c9b4a0376516b19cce68af8eab1952136faa1dfd28f585923a8"
Sep 09 21:42:39.354693 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.354653    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x" podStartSLOduration=1.683566307 podStartE2EDuration="4.354634822s" totalImagesPullingTime="2.671068515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:39.353338351 +0000 UTC m=+252.250004360" watchObservedRunningTime="2026-09-09 21:42:39.354634822 +0000 UTC m=+252.251300832"
Sep 09 21:42:39.514408 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.514381    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-48qfx"]
Sep 09 21:42:39.517399 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.517380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.519628 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.519605    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 21:42:39.519738 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.519670    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 21:42:39.519892 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.519874    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 21:42:39.520003 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.519983    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 21:42:39.520138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.520124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jhfz7\""
Sep 09 21:42:39.527181 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.527161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:39.527518 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.527501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48qfx"]
Sep 09 21:42:39.582891 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.582863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:39.671736 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.671628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-crio-socket\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.671736 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.671697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-data-volume\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.671736 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.671718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.671736 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.671742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6wpt\" (UniqueName: \"kubernetes.io/projected/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-api-access-t6wpt\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.672040 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.671842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.773249 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.773211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-data-volume\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.773249 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.773247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.773447 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.773272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6wpt\" (UniqueName: \"kubernetes.io/projected/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-api-access-t6wpt\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.773447 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:39.773386    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:39.773515 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:39.773455    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls podName:9de15aa3-26a6-4d46-89fe-c08d0b6a853e nodeName:}" failed. No retries permitted until 2026-09-09 21:42:40.273434512 +0000 UTC m=+253.170100514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48qfx" (UID: "9de15aa3-26a6-4d46-89fe-c08d0b6a853e") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:39.773515 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.773505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.773594 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:39.773580    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:42:39.773634 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.773618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-crio-socket\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.773669 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:39.773636    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm podName:9de15aa3-26a6-4d46-89fe-c08d0b6a853e nodeName:}" failed. No retries permitted until 2026-09-09 21:42:40.27362477 +0000 UTC m=+253.170290755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48qfx" (UID: "9de15aa3-26a6-4d46-89fe-c08d0b6a853e") : configmap "kube-rbac-proxy" not found
Sep 09 21:42:39.773669 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.773646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-data-volume\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.773728 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.773674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-crio-socket\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.783606 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.783568    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:39.783753 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.783730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6wpt\" (UniqueName: \"kubernetes.io/projected/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-api-access-t6wpt\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:39.883510 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:39.883475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:40.186643 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.186610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:40.277589 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.277551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:40.277798 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.277626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:40.277798 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:40.277701    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:42:40.277798 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:40.277724    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:40.277933 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:40.277802    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm podName:9de15aa3-26a6-4d46-89fe-c08d0b6a853e nodeName:}" failed. No retries permitted until 2026-09-09 21:42:41.277759895 +0000 UTC m=+254.174425895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48qfx" (UID: "9de15aa3-26a6-4d46-89fe-c08d0b6a853e") : configmap "kube-rbac-proxy" not found
Sep 09 21:42:40.277933 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:40.277824    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls podName:9de15aa3-26a6-4d46-89fe-c08d0b6a853e nodeName:}" failed. No retries permitted until 2026-09-09 21:42:41.277815554 +0000 UTC m=+254.174481553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48qfx" (UID: "9de15aa3-26a6-4d46-89fe-c08d0b6a853e") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:40.289959 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.289934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:40.349553 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.349513    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:42:40.349969 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.349955    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/0.log"
Sep 09 21:42:40.350018 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.349993    2563 generic.go:356] "Generic (PLEG): container finished" podID="448a39a8-aab5-4696-b385-2169e6ee4a6e" containerID="d9fbdc2a99809b310ac8970f9c0c94d6f3dfe61344ef039b8fb2624afc397779" exitCode=255
Sep 09 21:42:40.350110 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.350083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z2v2b" event={"ID":"448a39a8-aab5-4696-b385-2169e6ee4a6e","Type":"ContainerDied","Data":"d9fbdc2a99809b310ac8970f9c0c94d6f3dfe61344ef039b8fb2624afc397779"}
Sep 09 21:42:40.350208 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.350171    2563 scope.go:117] "RemoveContainer" containerID="cd55c3a46d022c9b4a0376516b19cce68af8eab1952136faa1dfd28f585923a8"
Sep 09 21:42:40.350353 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.350335    2563 scope.go:117] "RemoveContainer" containerID="d9fbdc2a99809b310ac8970f9c0c94d6f3dfe61344ef039b8fb2624afc397779"
Sep 09 21:42:40.350523 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:40.350504    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-z2v2b_openshift-console-operator(448a39a8-aab5-4696-b385-2169e6ee4a6e)\"" pod="openshift-console-operator/console-operator-85b454874b-z2v2b" podUID="448a39a8-aab5-4696-b385-2169e6ee4a6e"
Sep 09 21:42:40.369316 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.369292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:40.420456 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.420421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:40.481857 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:40.481754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:41.280753 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:41.280725    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-68g9z_d7198ae4-9d2d-41cb-892c-90e8f729c20e/dns-node-resolver/0.log"
Sep 09 21:42:41.287085 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:41.287059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:41.287188 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:41.287143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:41.287245 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:41.287207    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:41.287245 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:41.287234    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:42:41.287345 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:41.287274    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls podName:9de15aa3-26a6-4d46-89fe-c08d0b6a853e nodeName:}" failed. No retries permitted until 2026-09-09 21:42:43.287256532 +0000 UTC m=+256.183922523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48qfx" (UID: "9de15aa3-26a6-4d46-89fe-c08d0b6a853e") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:41.287345 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:41.287294    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm podName:9de15aa3-26a6-4d46-89fe-c08d0b6a853e nodeName:}" failed. No retries permitted until 2026-09-09 21:42:43.28728209 +0000 UTC m=+256.183948117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48qfx" (UID: "9de15aa3-26a6-4d46-89fe-c08d0b6a853e") : configmap "kube-rbac-proxy" not found
Sep 09 21:42:41.355414 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:41.355389    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:42:41.355732 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:41.355717    2563 scope.go:117] "RemoveContainer" containerID="d9fbdc2a99809b310ac8970f9c0c94d6f3dfe61344ef039b8fb2624afc397779"
Sep 09 21:42:41.355897 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:41.355879    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-z2v2b_openshift-console-operator(448a39a8-aab5-4696-b385-2169e6ee4a6e)\"" pod="openshift-console-operator/console-operator-85b454874b-z2v2b" podUID="448a39a8-aab5-4696-b385-2169e6ee4a6e"
Sep 09 21:42:42.067650 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.067616    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-bpmhs"]
Sep 09 21:42:42.071894 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.071877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.074272 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.074254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 21:42:42.074379 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.074253    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 21:42:42.074436 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.074406    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 21:42:42.075452 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.075440    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 21:42:42.075594 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.075573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tzdp4\""
Sep 09 21:42:42.077391 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.077371    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-bpmhs"]
Sep 09 21:42:42.098935 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.098911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:42.154072 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.154039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:42.195922 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.195891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59hds\" (UniqueName: \"kubernetes.io/projected/e60161b3-4abb-48ca-a389-568f16e579c5-kube-api-access-59hds\") pod \"service-ca-75d99c484-bpmhs\" (UID: \"e60161b3-4abb-48ca-a389-568f16e579c5\") " pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.196026 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.195982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e60161b3-4abb-48ca-a389-568f16e579c5-signing-key\") pod \"service-ca-75d99c484-bpmhs\" (UID: \"e60161b3-4abb-48ca-a389-568f16e579c5\") " pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.196076 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.196042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e60161b3-4abb-48ca-a389-568f16e579c5-signing-cabundle\") pod \"service-ca-75d99c484-bpmhs\" (UID: \"e60161b3-4abb-48ca-a389-568f16e579c5\") " pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.296630 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.296594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e60161b3-4abb-48ca-a389-568f16e579c5-signing-cabundle\") pod \"service-ca-75d99c484-bpmhs\" (UID: \"e60161b3-4abb-48ca-a389-568f16e579c5\") " pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.297028 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.296670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59hds\" (UniqueName: \"kubernetes.io/projected/e60161b3-4abb-48ca-a389-568f16e579c5-kube-api-access-59hds\") pod \"service-ca-75d99c484-bpmhs\" (UID: \"e60161b3-4abb-48ca-a389-568f16e579c5\") " pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.297028 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.296832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e60161b3-4abb-48ca-a389-568f16e579c5-signing-key\") pod \"service-ca-75d99c484-bpmhs\" (UID: \"e60161b3-4abb-48ca-a389-568f16e579c5\") " pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.297320 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.297302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e60161b3-4abb-48ca-a389-568f16e579c5-signing-cabundle\") pod \"service-ca-75d99c484-bpmhs\" (UID: \"e60161b3-4abb-48ca-a389-568f16e579c5\") " pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.299239 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.299223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e60161b3-4abb-48ca-a389-568f16e579c5-signing-key\") pod \"service-ca-75d99c484-bpmhs\" (UID: \"e60161b3-4abb-48ca-a389-568f16e579c5\") " pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.305107 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.305087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59hds\" (UniqueName: \"kubernetes.io/projected/e60161b3-4abb-48ca-a389-568f16e579c5-kube-api-access-59hds\") pod \"service-ca-75d99c484-bpmhs\" (UID: \"e60161b3-4abb-48ca-a389-568f16e579c5\") " pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.381709 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.381624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-bpmhs"
Sep 09 21:42:42.655088 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.655010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-bpmhs"]
Sep 09 21:42:42.658015 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:42:42.657987    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60161b3_4abb_48ca_a389_568f16e579c5.slice/crio-bc2b506603977a704aef1f826ad7d09318add04a2a1ae0f0720cd8b95a97c639 WatchSource:0}: Error finding container bc2b506603977a704aef1f826ad7d09318add04a2a1ae0f0720cd8b95a97c639: Status 404 returned error can't find the container with id bc2b506603977a704aef1f826ad7d09318add04a2a1ae0f0720cd8b95a97c639
Sep 09 21:42:42.687014 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.686996    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tlfbl_715cecb6-2493-4cd4-84b8-6eb84e62fb1b/node-ca/0.log"
Sep 09 21:42:42.903111 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.903078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:42.903111 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.903128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:42.903493 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:42.903177    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:42.903493 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:42.903232    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 21:42:42.903493 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:42.903250    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls podName:6ab3349e-53f6-462b-a888-bff0d463fd2f nodeName:}" failed. No retries permitted until 2026-09-09 21:42:50.903235147 +0000 UTC m=+263.799901138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8pcxl" (UID: "6ab3349e-53f6-462b-a888-bff0d463fd2f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:42.903493 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:42.903268    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls podName:99f5bc95-24e1-4520-b21f-57edc4133ff8 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:50.903257014 +0000 UTC m=+263.799923005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hdssz" (UID: "99f5bc95-24e1-4520-b21f-57edc4133ff8") : secret "samples-operator-tls" not found
Sep 09 21:42:42.956906 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:42.956825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:43.306147 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:43.306115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:43.306488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:43.306205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:43.306488 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:43.306266    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:43.306488 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:43.306330    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls podName:9de15aa3-26a6-4d46-89fe-c08d0b6a853e nodeName:}" failed. No retries permitted until 2026-09-09 21:42:47.306315427 +0000 UTC m=+260.202981417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48qfx" (UID: "9de15aa3-26a6-4d46-89fe-c08d0b6a853e") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:43.306690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:43.306673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:43.363178 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:43.363146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-bpmhs" event={"ID":"e60161b3-4abb-48ca-a389-568f16e579c5","Type":"ContainerStarted","Data":"6abe53889333a9467152b43055f4e30453991cc0f7ed3c5df1fc945c0c645d45"}
Sep 09 21:42:43.363336 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:43.363195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-bpmhs" event={"ID":"e60161b3-4abb-48ca-a389-568f16e579c5","Type":"ContainerStarted","Data":"bc2b506603977a704aef1f826ad7d09318add04a2a1ae0f0720cd8b95a97c639"}
Sep 09 21:42:43.379039 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:43.378988    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-bpmhs" podStartSLOduration=1.378972923 podStartE2EDuration="1.378972923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:43.377481604 +0000 UTC m=+256.274147605" watchObservedRunningTime="2026-09-09 21:42:43.378972923 +0000 UTC m=+256.275638993"
Sep 09 21:42:43.484266 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:43.484234    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n6cbw_0b8f9430-2b08-481b-9d13-4290193c0092/serve-healthcheck-canary/0.log"
Sep 09 21:42:44.049783 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:44.049745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:44.460362 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:44.460268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:44.559574 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:44.559525    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:44.663668 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:44.663611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:44.763877 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:44.763843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:44.864641 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:44.864598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:44.965298 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:44.965262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:45.225942 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:45.225866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:45.391657 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:45.391620    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:45.391845 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:45.391705    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:45.392062 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:45.392049    2563 scope.go:117] "RemoveContainer" containerID="d9fbdc2a99809b310ac8970f9c0c94d6f3dfe61344ef039b8fb2624afc397779"
Sep 09 21:42:45.392210 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:45.392191    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-z2v2b_openshift-console-operator(448a39a8-aab5-4696-b385-2169e6ee4a6e)\"" pod="openshift-console-operator/console-operator-85b454874b-z2v2b" podUID="448a39a8-aab5-4696-b385-2169e6ee4a6e"
Sep 09 21:42:45.490785 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:45.490684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:46.221065 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:46.221030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:46.615847 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:46.615809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:46.821742 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:46.821700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:47.023679 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:47.023647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:47.123998 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:47.123965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:47.345587 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:47.345489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:47.345800 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:47.345639    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:47.345800 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:47.345712    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls podName:9de15aa3-26a6-4d46-89fe-c08d0b6a853e nodeName:}" failed. No retries permitted until 2026-09-09 21:42:55.34569102 +0000 UTC m=+268.242357007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48qfx" (UID: "9de15aa3-26a6-4d46-89fe-c08d0b6a853e") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:47.424647 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:47.424610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:47.929110 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:47.929057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:47.982915 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:47.982885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:48.035880 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:48.035843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:48.426195 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:48.426162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:48.821919 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:48.821889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:49.221285 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:49.221199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:49.423157 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:49.423129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:49.618650 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:49.618612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:49.847216 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:49.847185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:50.533859 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:50.533814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:50.817663 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:50.817573    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:50.973889 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:50.973849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:42:50.974045 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:50.973901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:50.974045 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:50.973991    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:50.974118 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:42:50.974067    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls podName:6ab3349e-53f6-462b-a888-bff0d463fd2f nodeName:}" failed. No retries permitted until 2026-09-09 21:43:06.974054041 +0000 UTC m=+279.870720030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8pcxl" (UID: "6ab3349e-53f6-462b-a888-bff0d463fd2f") : secret "cluster-monitoring-operator-tls" not found
Sep 09 21:42:50.976442 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:50.976420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99f5bc95-24e1-4520-b21f-57edc4133ff8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hdssz\" (UID: \"99f5bc95-24e1-4520-b21f-57edc4133ff8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:51.008326 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:51.008291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"
Sep 09 21:42:51.018983 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:51.018959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:51.221589 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:51.221502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:51.280978 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:51.280947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz"]
Sep 09 21:42:51.389601 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:51.389569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz" event={"ID":"99f5bc95-24e1-4520-b21f-57edc4133ff8","Type":"ContainerStarted","Data":"fa8f87951925973425dcd46f80db15a282d7b83b8486c5e203767ddec5529077"}
Sep 09 21:42:51.424351 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:51.424320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:51.619104 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:51.619071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:52.621595 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:52.621562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:52.820306 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:52.820272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:53.218095 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:53.218050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:53.399676 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:53.399641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz" event={"ID":"99f5bc95-24e1-4520-b21f-57edc4133ff8","Type":"ContainerStarted","Data":"fc50d2c5cf62c2bcf09d5d7a19ca207d6637e94c24a7edb3b20f556f72211e08"}
Sep 09 21:42:53.418846 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:53.418782    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:53.619709 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:53.619671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:54.404060 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:54.404028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz" event={"ID":"99f5bc95-24e1-4520-b21f-57edc4133ff8","Type":"ContainerStarted","Data":"d8a9a2bb1f80975981f975b7fa4d18f9cf2bacf0db481f1032b7a96a966d80f8"}
Sep 09 21:42:54.420041 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:54.419990    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hdssz" podStartSLOduration=17.450351362 podStartE2EDuration="19.419974391s" totalImagesPullingTime="1.969623029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:54.419141099 +0000 UTC m=+267.315807106" watchObservedRunningTime="2026-09-09 21:42:54.419974391 +0000 UTC m=+267.316640398"
Sep 09 21:42:54.424258 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:54.424236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:54.621741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:54.621707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:54.823333 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:54.823296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:55.303395 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:55.303363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:55.413154 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:55.413126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:55.415500 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:55.415466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9de15aa3-26a6-4d46-89fe-c08d0b6a853e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48qfx\" (UID: \"9de15aa3-26a6-4d46-89fe-c08d0b6a853e\") " pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:55.426434 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:55.426409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48qfx"
Sep 09 21:42:55.695072 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:55.694991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48qfx"]
Sep 09 21:42:55.697740 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:42:55.697711    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de15aa3_26a6_4d46_89fe_c08d0b6a853e.slice/crio-70d9e576d5163ad8eb9844a8bc9951d3368b2b8d0bbb4f8684ae8de1e08f320a WatchSource:0}: Error finding container 70d9e576d5163ad8eb9844a8bc9951d3368b2b8d0bbb4f8684ae8de1e08f320a: Status 404 returned error can't find the container with id 70d9e576d5163ad8eb9844a8bc9951d3368b2b8d0bbb4f8684ae8de1e08f320a
Sep 09 21:42:56.410961 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:56.410923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48qfx" event={"ID":"9de15aa3-26a6-4d46-89fe-c08d0b6a853e","Type":"ContainerStarted","Data":"f2896f09640a510d36c073fc79cc53862b61da07d2cb528366548910cfdb2c23"}
Sep 09 21:42:56.411111 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:56.410986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48qfx" event={"ID":"9de15aa3-26a6-4d46-89fe-c08d0b6a853e","Type":"ContainerStarted","Data":"70d9e576d5163ad8eb9844a8bc9951d3368b2b8d0bbb4f8684ae8de1e08f320a"}
Sep 09 21:42:56.600191 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:56.600170    2563 scope.go:117] "RemoveContainer" containerID="d9fbdc2a99809b310ac8970f9c0c94d6f3dfe61344ef039b8fb2624afc397779"
Sep 09 21:42:57.416839 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:57.416811    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:42:57.417110 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:57.416944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z2v2b" event={"ID":"448a39a8-aab5-4696-b385-2169e6ee4a6e","Type":"ContainerStarted","Data":"0503b2c6bd2a649254a028fdfe9bded7b74394f124353794b63fa75a691e34f5"}
Sep 09 21:42:57.417482 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:57.417421    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:57.419533 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:57.419509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48qfx" event={"ID":"9de15aa3-26a6-4d46-89fe-c08d0b6a853e","Type":"ContainerStarted","Data":"db3e783b0c9cba916d806b244e5af0de6c7a4a562e4655968b8558a368e655e2"}
Sep 09 21:42:57.455637 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:57.455578    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-z2v2b" podStartSLOduration=19.686547095 podStartE2EDuration="22.4555632s" totalImagesPullingTime="2.769016105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:57.454884768 +0000 UTC m=+270.351550777" watchObservedRunningTime="2026-09-09 21:42:57.4555632 +0000 UTC m=+270.352229207"
Sep 09 21:42:57.503660 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:57.503630    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-z2v2b"
Sep 09 21:42:57.682328 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:57.682252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:57.783418 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:57.783383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:57.841073 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:57.841035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:57.977063 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:57.976990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:58.354938 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:58.354904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:58.424160 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:58.424125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48qfx" event={"ID":"9de15aa3-26a6-4d46-89fe-c08d0b6a853e","Type":"ContainerStarted","Data":"c62a67e912eb3be57ba0f1c1bfd6e15dab07f30c0e576c84cd28b2f263a80bf1"}
Sep 09 21:42:58.443081 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:58.443033    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-48qfx" podStartSLOduration=17.313329324 podStartE2EDuration="19.44301759s" totalImagesPullingTime="2.129688266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:42:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:58.44118253 +0000 UTC m=+271.337848537" watchObservedRunningTime="2026-09-09 21:42:58.44301759 +0000 UTC m=+271.339683598"
Sep 09 21:42:58.446306 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:58.446276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:58.559751 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:58.559714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:58.619104 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:58.619019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:58.764533 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:58.764494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:58.981571 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:58.981484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:59.031145 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:59.031115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:59.362398 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:59.362370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:59.577372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:59.577337    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:59.678262 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:42:59.678184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:00.458276 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.458242    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-cwvcs"]
Sep 09 21:43:00.461220 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.461201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-cwvcs"
Sep 09 21:43:00.463497 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.463453    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 21:43:00.463614 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.463539    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 21:43:00.464123 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.464110    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-slndv\""
Sep 09 21:43:00.474953 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.474924    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-cwvcs"]
Sep 09 21:43:00.497047 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.497016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:00.550719 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.550680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmc92\" (UniqueName: \"kubernetes.io/projected/332f31a4-50a6-4169-9b12-c76283c2f7d4-kube-api-access-wmc92\") pod \"downloads-85d98c998f-cwvcs\" (UID: \"332f31a4-50a6-4169-9b12-c76283c2f7d4\") " pod="openshift-console/downloads-85d98c998f-cwvcs"
Sep 09 21:43:00.589908 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.589878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:00.651792 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.651743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmc92\" (UniqueName: \"kubernetes.io/projected/332f31a4-50a6-4169-9b12-c76283c2f7d4-kube-api-access-wmc92\") pod \"downloads-85d98c998f-cwvcs\" (UID: \"332f31a4-50a6-4169-9b12-c76283c2f7d4\") " pod="openshift-console/downloads-85d98c998f-cwvcs"
Sep 09 21:43:00.661318 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.661285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmc92\" (UniqueName: \"kubernetes.io/projected/332f31a4-50a6-4169-9b12-c76283c2f7d4-kube-api-access-wmc92\") pod \"downloads-85d98c998f-cwvcs\" (UID: \"332f31a4-50a6-4169-9b12-c76283c2f7d4\") " pod="openshift-console/downloads-85d98c998f-cwvcs"
Sep 09 21:43:00.689613 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.689585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:00.770607 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:00.770574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-cwvcs"
Sep 09 21:43:01.041996 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:01.041923    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-cwvcs"]
Sep 09 21:43:01.044693 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:43:01.044667    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332f31a4_50a6_4169_9b12_c76283c2f7d4.slice/crio-f99f2acab7f7a0ac2a0fa4c19a297dd81ea54c17486969e5038c8d86295d9fde WatchSource:0}: Error finding container f99f2acab7f7a0ac2a0fa4c19a297dd81ea54c17486969e5038c8d86295d9fde: Status 404 returned error can't find the container with id f99f2acab7f7a0ac2a0fa4c19a297dd81ea54c17486969e5038c8d86295d9fde
Sep 09 21:43:01.185213 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:01.185174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:01.256225 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:01.256198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:01.434752 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:01.434670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-cwvcs" event={"ID":"332f31a4-50a6-4169-9b12-c76283c2f7d4","Type":"ContainerStarted","Data":"f99f2acab7f7a0ac2a0fa4c19a297dd81ea54c17486969e5038c8d86295d9fde"}
Sep 09 21:43:02.017004 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:02.016973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:02.068605 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:02.068565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:07.005227 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:07.005192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:43:07.008109 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:07.008082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ab3349e-53f6-462b-a888-bff0d463fd2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8pcxl\" (UID: \"6ab3349e-53f6-462b-a888-bff0d463fd2f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:43:07.203547 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:07.203506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"
Sep 09 21:43:07.480755 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:07.480637    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl"]
Sep 09 21:43:07.483439 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:43:07.483408    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab3349e_53f6_462b_a888_bff0d463fd2f.slice/crio-57364c16e8392dca023d26ca1f2fc81c0e3b57eb07aee5d23f28dae1476259f0 WatchSource:0}: Error finding container 57364c16e8392dca023d26ca1f2fc81c0e3b57eb07aee5d23f28dae1476259f0: Status 404 returned error can't find the container with id 57364c16e8392dca023d26ca1f2fc81c0e3b57eb07aee5d23f28dae1476259f0
Sep 09 21:43:08.461238 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:08.461203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl" event={"ID":"6ab3349e-53f6-462b-a888-bff0d463fd2f","Type":"ContainerStarted","Data":"57364c16e8392dca023d26ca1f2fc81c0e3b57eb07aee5d23f28dae1476259f0"}
Sep 09 21:43:09.829347 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:09.829235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:09.930213 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:09.930178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:09.982265 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:09.982230    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52"]
Sep 09 21:43:09.985781 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:09.985735    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52"
Sep 09 21:43:09.988177 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:09.988155    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 21:43:09.988309 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:09.988242    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zxdbg\""
Sep 09 21:43:10.001609 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:10.001578    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52"]
Sep 09 21:43:10.005570 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:10.005531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:10.034276 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:10.034240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e9842591-f7c3-4a3c-be71-d425623e63a2-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-f2p52\" (UID: \"e9842591-f7c3-4a3c-be71-d425623e63a2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52"
Sep 09 21:43:10.135398 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:10.135309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e9842591-f7c3-4a3c-be71-d425623e63a2-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-f2p52\" (UID: \"e9842591-f7c3-4a3c-be71-d425623e63a2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52"
Sep 09 21:43:10.138436 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:10.138407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e9842591-f7c3-4a3c-be71-d425623e63a2-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-f2p52\" (UID: \"e9842591-f7c3-4a3c-be71-d425623e63a2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52"
Sep 09 21:43:10.163503 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:10.163459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:10.300379 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:10.300340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52"
Sep 09 21:43:10.470021 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:10.469938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl" event={"ID":"6ab3349e-53f6-462b-a888-bff0d463fd2f","Type":"ContainerStarted","Data":"a57861d37cee500c2bae218aa6c65dab486f49974fdc5637497fdb961b96bfe0"}
Sep 09 21:43:10.487423 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:10.487368    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8pcxl" podStartSLOduration=33.463344059 podStartE2EDuration="35.487352226s" totalImagesPullingTime="2.024008167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:10.486327621 +0000 UTC m=+283.382993630" watchObservedRunningTime="2026-09-09 21:43:10.487352226 +0000 UTC m=+283.384018234"
Sep 09 21:43:10.582071 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:10.582027    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52"]
Sep 09 21:43:12.988096 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:12.988063    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68df84699f-9jh42"]
Sep 09 21:43:12.992343 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:12.992322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:12.994852 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:12.994829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hzl8x\""
Sep 09 21:43:12.994971 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:12.994833    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 21:43:12.994971 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:12.994836    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 21:43:12.994971 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:12.994838    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 21:43:12.995741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:12.995724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 21:43:12.996218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:12.996200    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 21:43:13.001727 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.001599    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 21:43:13.001727 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.001637    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68df84699f-9jh42"]
Sep 09 21:43:13.060274 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.060238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-config\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.060274 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.060277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-trusted-ca-bundle\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.060514 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.060369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-oauth-serving-cert\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.060514 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.060392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-service-ca\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.060514 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.060416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whz5h\" (UniqueName: \"kubernetes.io/projected/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-kube-api-access-whz5h\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.060514 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.060454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-oauth-config\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.060705 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.060522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-serving-cert\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.161326 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.161287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-config\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.161326 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.161326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-trusted-ca-bundle\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.161599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.161377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-oauth-serving-cert\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.161599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.161402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-service-ca\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.161599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.161428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whz5h\" (UniqueName: \"kubernetes.io/projected/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-kube-api-access-whz5h\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.161599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.161471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-oauth-config\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.161599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.161508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-serving-cert\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.162173 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.162143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-oauth-serving-cert\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.162263 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.162177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-config\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.162263 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.162202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-service-ca\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.162677 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.162653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-trusted-ca-bundle\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.164314 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.164293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-serving-cert\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.164414 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.164330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-oauth-config\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.169203 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.169180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whz5h\" (UniqueName: \"kubernetes.io/projected/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-kube-api-access-whz5h\") pod \"console-68df84699f-9jh42\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") " pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.305067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.305019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:13.382931 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:13.382898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:15.122744 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:43:15.122699    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp" podUID="f742f48a-2f95-4121-adaf-4a1f023498df"
Sep 09 21:43:15.123161 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:43:15.122735    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zh6lr" podUID="888f9b50-262e-4525-a9f2-3857190be163"
Sep 09 21:43:15.491857 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:15.491762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:43:15.491857 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:15.491822    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:43:17.027150 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:17.027110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:17.127717 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:17.127681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:17.228028 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:17.227996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:17.277555 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:17.277437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:17.509214 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:43:17.509180    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9842591_f7c3_4a3c_be71_d425623e63a2.slice/crio-93ab4c8c0b35fa1a1ec07b89348225c3a592b3d3e76902ad9b99eaa2d172e251 WatchSource:0}: Error finding container 93ab4c8c0b35fa1a1ec07b89348225c3a592b3d3e76902ad9b99eaa2d172e251: Status 404 returned error can't find the container with id 93ab4c8c0b35fa1a1ec07b89348225c3a592b3d3e76902ad9b99eaa2d172e251
Sep 09 21:43:17.776464 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:17.776428    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68df84699f-9jh42"]
Sep 09 21:43:17.779815 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:43:17.779764    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b6c792_7dc5_42b8_b0e4_3d74481c150f.slice/crio-9de0abc3dc2b16be0b5d7e555dc01c1f515453b4636f619c59c0a92747a19802 WatchSource:0}: Error finding container 9de0abc3dc2b16be0b5d7e555dc01c1f515453b4636f619c59c0a92747a19802: Status 404 returned error can't find the container with id 9de0abc3dc2b16be0b5d7e555dc01c1f515453b4636f619c59c0a92747a19802
Sep 09 21:43:18.511245 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.511176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-cwvcs" event={"ID":"332f31a4-50a6-4169-9b12-c76283c2f7d4","Type":"ContainerStarted","Data":"2d4bfc46741a619b0037e1db60f625382fc8bb4a86c63fe383cc5ec0d284cf10"}
Sep 09 21:43:18.512012 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.511370    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-cwvcs"
Sep 09 21:43:18.513652 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.513622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52" event={"ID":"e9842591-f7c3-4a3c-be71-d425623e63a2","Type":"ContainerStarted","Data":"93ab4c8c0b35fa1a1ec07b89348225c3a592b3d3e76902ad9b99eaa2d172e251"}
Sep 09 21:43:18.515419 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.515396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df84699f-9jh42" event={"ID":"f1b6c792-7dc5-42b8-b0e4-3d74481c150f","Type":"ContainerStarted","Data":"9de0abc3dc2b16be0b5d7e555dc01c1f515453b4636f619c59c0a92747a19802"}
Sep 09 21:43:18.523632 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.523583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-cwvcs"
Sep 09 21:43:18.530793 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.530508    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-cwvcs" podStartSLOduration=1.975898746 podStartE2EDuration="18.530489442s" totalImagesPullingTime="16.554590696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:18.528917813 +0000 UTC m=+291.425583847" watchObservedRunningTime="2026-09-09 21:43:18.530489442 +0000 UTC m=+291.427155450"
Sep 09 21:43:18.916617 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.916014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:43:18.916617 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.916103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:43:18.916617 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.916166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:43:18.925312 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.922882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f742f48a-2f95-4121-adaf-4a1f023498df-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-vs5kp\" (UID: \"f742f48a-2f95-4121-adaf-4a1f023498df\") " pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:43:18.926148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.926076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/888f9b50-262e-4525-a9f2-3857190be163-metrics-tls\") pod \"dns-default-zh6lr\" (UID: \"888f9b50-262e-4525-a9f2-3857190be163\") " pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:43:18.926792 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:18.926695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"image-registry-b6d496f4d-s986w\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") " pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:43:19.094743 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:19.094690    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xrjln\""
Sep 09 21:43:19.095083 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:19.095058    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8h77q\""
Sep 09 21:43:19.102517 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:19.102481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp"
Sep 09 21:43:19.102806 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:19.102784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:43:19.203163 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:19.202930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xvgqt\""
Sep 09 21:43:19.210991 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:19.210949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:43:19.619452 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:19.619375    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-vs5kp"]
Sep 09 21:43:19.625262 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:43:19.625233    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf742f48a_2f95_4121_adaf_4a1f023498df.slice/crio-4faf40c99f90f8ef23c08afe0b21a27e19b0edbd148988652c102c7bdd0179ef WatchSource:0}: Error finding container 4faf40c99f90f8ef23c08afe0b21a27e19b0edbd148988652c102c7bdd0179ef: Status 404 returned error can't find the container with id 4faf40c99f90f8ef23c08afe0b21a27e19b0edbd148988652c102c7bdd0179ef
Sep 09 21:43:19.629382 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:19.629360    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zh6lr"]
Sep 09 21:43:19.635644 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:19.635567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b6d496f4d-s986w"]
Sep 09 21:43:19.639080 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:43:19.639047    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9481e7b_af31_48ec_9504_a706c8d02cbd.slice/crio-7e474bb553977a07f2b244c24722d90a0e1a61e3710fb0d64ecdbaeb5e6c9c36 WatchSource:0}: Error finding container 7e474bb553977a07f2b244c24722d90a0e1a61e3710fb0d64ecdbaeb5e6c9c36: Status 404 returned error can't find the container with id 7e474bb553977a07f2b244c24722d90a0e1a61e3710fb0d64ecdbaeb5e6c9c36
Sep 09 21:43:20.528903 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.528721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp" event={"ID":"f742f48a-2f95-4121-adaf-4a1f023498df","Type":"ContainerStarted","Data":"4faf40c99f90f8ef23c08afe0b21a27e19b0edbd148988652c102c7bdd0179ef"}
Sep 09 21:43:20.531632 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.531571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52" event={"ID":"e9842591-f7c3-4a3c-be71-d425623e63a2","Type":"ContainerStarted","Data":"9ed96f07bbbd38df563f4386aa898e7a67157f72c2280f94073c103ecba7c094"}
Sep 09 21:43:20.532389 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.532368    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52"
Sep 09 21:43:20.536617 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.536585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b6d496f4d-s986w" event={"ID":"b9481e7b-af31-48ec-9504-a706c8d02cbd","Type":"ContainerStarted","Data":"865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b"}
Sep 09 21:43:20.536836 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.536654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b6d496f4d-s986w" event={"ID":"b9481e7b-af31-48ec-9504-a706c8d02cbd","Type":"ContainerStarted","Data":"7e474bb553977a07f2b244c24722d90a0e1a61e3710fb0d64ecdbaeb5e6c9c36"}
Sep 09 21:43:20.537064 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.537026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:43:20.541803 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.539343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52"
Sep 09 21:43:20.544121 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.544067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zh6lr" event={"ID":"888f9b50-262e-4525-a9f2-3857190be163","Type":"ContainerStarted","Data":"d659641eb9443ba1e8a93d230859449bfab118fddb0e8ddf0d7ef4c0e0107060"}
Sep 09 21:43:20.555333 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.554935    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-f2p52" podStartSLOduration=9.598147746 podStartE2EDuration="11.554917637s" totalImagesPullingTime="1.956769891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:20.553118938 +0000 UTC m=+293.449784947" watchObservedRunningTime="2026-09-09 21:43:20.554917637 +0000 UTC m=+293.451583648"
Sep 09 21:43:20.573857 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.573638    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b6d496f4d-s986w" podStartSLOduration=293.573619067 podStartE2EDuration="4m53.573619067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:20.571647305 +0000 UTC m=+293.468313316" watchObservedRunningTime="2026-09-09 21:43:20.573619067 +0000 UTC m=+293.470285078"
Sep 09 21:43:20.997713 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:20.997669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:21.034716 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.034360    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-tcx77"]
Sep 09 21:43:21.055851 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.055750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-tcx77"]
Sep 09 21:43:21.055851 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.055825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.058695 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.058537    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wgzpl\""
Sep 09 21:43:21.058695 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.058555    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 21:43:21.058903 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.058791    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 21:43:21.059544 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.059523    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 21:43:21.075357 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.075264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:21.136833 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.136640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.136833 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.136687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9frm\" (UniqueName: \"kubernetes.io/projected/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-kube-api-access-z9frm\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.136833 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.136742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.136833 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.136791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.239608 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.237974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.239608 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.238023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9frm\" (UniqueName: \"kubernetes.io/projected/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-kube-api-access-z9frm\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.239608 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.238071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.239608 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.238097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.239608 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.239566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.240022 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:43:21.239618    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 09 21:43:21.240022 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:43:21.239679    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-prometheus-operator-tls podName:85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1 nodeName:}" failed. No retries permitted until 2026-09-09 21:43:21.739663442 +0000 UTC m=+294.636329429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-tcx77" (UID: "85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1") : secret "prometheus-operator-tls" not found
Sep 09 21:43:21.242843 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.242794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.253726 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.253656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9frm\" (UniqueName: \"kubernetes.io/projected/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-kube-api-access-z9frm\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.344722 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.344676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:21.744124 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.744082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.747240 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.747134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-tcx77\" (UID: \"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:21.978597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:21.978503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77"
Sep 09 21:43:22.098025 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:22.097989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:22.199384 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:22.199347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:24.006482 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.006446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-tcx77"]
Sep 09 21:43:24.014394 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:43:24.014348    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f25c3d_9d3e_4c3f_bf65_5f6b70ab74c1.slice/crio-7f664791274dc27f590849f29ff1087d9806444751d7e5a3ec0749b6dbe6659a WatchSource:0}: Error finding container 7f664791274dc27f590849f29ff1087d9806444751d7e5a3ec0749b6dbe6659a: Status 404 returned error can't find the container with id 7f664791274dc27f590849f29ff1087d9806444751d7e5a3ec0749b6dbe6659a
Sep 09 21:43:24.561622 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.561476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zh6lr" event={"ID":"888f9b50-262e-4525-a9f2-3857190be163","Type":"ContainerStarted","Data":"d92a93fa18f05d9d9bb4614c5b7521f18ee7db0a6c950746cf9a67a0258cefbc"}
Sep 09 21:43:24.561622 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.561553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zh6lr" event={"ID":"888f9b50-262e-4525-a9f2-3857190be163","Type":"ContainerStarted","Data":"59ed5addcbf064094d08840ad173672acb44bef7de3a75a529f1aa7b6c61fc50"}
Sep 09 21:43:24.561622 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.561623    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:43:24.563707 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.563675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp" event={"ID":"f742f48a-2f95-4121-adaf-4a1f023498df","Type":"ContainerStarted","Data":"23db40c1cd2cab7d0783f7daed26fb23a664349b35ab5d1b37fe2fb0f439073f"}
Sep 09 21:43:24.565114 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.565090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77" event={"ID":"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1","Type":"ContainerStarted","Data":"7f664791274dc27f590849f29ff1087d9806444751d7e5a3ec0749b6dbe6659a"}
Sep 09 21:43:24.566807 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.566760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df84699f-9jh42" event={"ID":"f1b6c792-7dc5-42b8-b0e4-3d74481c150f","Type":"ContainerStarted","Data":"e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19"}
Sep 09 21:43:24.578042 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.577994    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zh6lr" podStartSLOduration=252.487475326 podStartE2EDuration="4m16.577976006s" totalImagesPullingTime="4.09050068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:24.576267693 +0000 UTC m=+297.472933705" watchObservedRunningTime="2026-09-09 21:43:24.577976006 +0000 UTC m=+297.474642014"
Sep 09 21:43:24.592300 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.592256    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-vs5kp" podStartSLOduration=281.487813718 podStartE2EDuration="4m45.592242526s" totalImagesPullingTime="4.104428808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:24.591233601 +0000 UTC m=+297.487899611" watchObservedRunningTime="2026-09-09 21:43:24.592242526 +0000 UTC m=+297.488908534"
Sep 09 21:43:24.607681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.607619    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68df84699f-9jh42" podStartSLOduration=6.663467902 podStartE2EDuration="12.607601s" totalImagesPullingTime="5.944133098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:24.606459391 +0000 UTC m=+297.503125413" watchObservedRunningTime="2026-09-09 21:43:24.607601 +0000 UTC m=+297.504267009"
Sep 09 21:43:24.705327 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.705293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:24.755239 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.755201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:24.812253 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:24.812096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:25.792669 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:25.792632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:25.843133 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:25.843093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:26.200140 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:26.199989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:26.303604 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:26.303564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:26.427022 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:26.426965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:26.577312 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:26.577277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77" event={"ID":"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1","Type":"ContainerStarted","Data":"d3760105525b6272b62186539a6d5eefc8c9f1fe432cfaeeedebb1111ebaadc7"}
Sep 09 21:43:26.577489 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:26.577337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77" event={"ID":"85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1","Type":"ContainerStarted","Data":"b8a836e21204324eb47cbf2f26a6f7e3c589721a11d83c889492410893120ff0"}
Sep 09 21:43:26.593923 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:26.593753    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-tcx77" podStartSLOduration=3.524216669 podStartE2EDuration="5.593733444s" totalImagesPullingTime="2.069516775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:26.59361846 +0000 UTC m=+299.490284468" watchObservedRunningTime="2026-09-09 21:43:26.593733444 +0000 UTC m=+299.490399454"
Sep 09 21:43:26.594806 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:26.594710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:26.692970 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:26.692937    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:27.482633 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:27.482605    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:43:27.483151 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:27.482844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:43:27.493005 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:27.492981    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 21:43:28.125997 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.125961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:28.341888 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.341846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:28.390056 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.389963    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ghzwq"]
Sep 09 21:43:28.409742 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.409718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.412386 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.412361    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-258df\""
Sep 09 21:43:28.412518 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.412373    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 21:43:28.412785 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.412750    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 21:43:28.413062 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.413047    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 21:43:28.424912 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.424890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:28.509583 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.509545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-tls\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.510067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.509625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-wtmp\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.510067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.509654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbg5c\" (UniqueName: \"kubernetes.io/projected/f7714c74-de4f-4360-a16a-5a68938045c6-kube-api-access-hbg5c\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.510067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.509687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7714c74-de4f-4360-a16a-5a68938045c6-root\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.510067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.509716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-textfile\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.510067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.509761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7714c74-de4f-4360-a16a-5a68938045c6-metrics-client-ca\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.510067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.509846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7714c74-de4f-4360-a16a-5a68938045c6-sys\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.510067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.509900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.510067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.509948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.618833 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.618799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-wtmp\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.618833 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.618845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbg5c\" (UniqueName: \"kubernetes.io/projected/f7714c74-de4f-4360-a16a-5a68938045c6-kube-api-access-hbg5c\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.619071 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.618882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7714c74-de4f-4360-a16a-5a68938045c6-root\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.619071 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.618909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-textfile\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.619071 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.618955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7714c74-de4f-4360-a16a-5a68938045c6-metrics-client-ca\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.619071 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.618984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7714c74-de4f-4360-a16a-5a68938045c6-sys\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.619071 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.619007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.619071 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.619038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.619349 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.619089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-tls\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.619349 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:43:28.619212    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 21:43:28.619349 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:43:28.619279    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-tls podName:f7714c74-de4f-4360-a16a-5a68938045c6 nodeName:}" failed. No retries permitted until 2026-09-09 21:43:29.119260481 +0000 UTC m=+302.015926473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-tls") pod "node-exporter-ghzwq" (UID: "f7714c74-de4f-4360-a16a-5a68938045c6") : secret "node-exporter-tls" not found
Sep 09 21:43:28.619632 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.619610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7714c74-de4f-4360-a16a-5a68938045c6-sys\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.619902 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.619883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7714c74-de4f-4360-a16a-5a68938045c6-root\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.620182 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.620156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-wtmp\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.620331 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.620203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.621427 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.620438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-textfile\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.621427 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.620853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7714c74-de4f-4360-a16a-5a68938045c6-metrics-client-ca\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.631754 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.631722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.641597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.641490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbg5c\" (UniqueName: \"kubernetes.io/projected/f7714c74-de4f-4360-a16a-5a68938045c6-kube-api-access-hbg5c\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:28.643378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.643334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:28.777961 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.777926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:28.903264 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:28.903181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:29.004208 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:29.004176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:29.056804 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:29.056757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:29.124508 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:29.124469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-tls\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:29.126969 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:29.126946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7714c74-de4f-4360-a16a-5a68938045c6-node-exporter-tls\") pod \"node-exporter-ghzwq\" (UID: \"f7714c74-de4f-4360-a16a-5a68938045c6\") " pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:29.320716 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:29.320682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ghzwq"
Sep 09 21:43:29.350359 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:29.350327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:29.356069 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:43:29.356045    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7714c74_de4f_4360_a16a_5a68938045c6.slice/crio-3d00dd65f85fc78fe0ca30a7915b811924bf16198841ec2dae53fc70f5ec4bec WatchSource:0}: Error finding container 3d00dd65f85fc78fe0ca30a7915b811924bf16198841ec2dae53fc70f5ec4bec: Status 404 returned error can't find the container with id 3d00dd65f85fc78fe0ca30a7915b811924bf16198841ec2dae53fc70f5ec4bec
Sep 09 21:43:29.358078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:29.358063    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:43:29.425523 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:29.425492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:29.542463 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:29.542425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:29.591021 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:29.590943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghzwq" event={"ID":"f7714c74-de4f-4360-a16a-5a68938045c6","Type":"ContainerStarted","Data":"3d00dd65f85fc78fe0ca30a7915b811924bf16198841ec2dae53fc70f5ec4bec"}
Sep 09 21:43:29.600986 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:43:29.600956    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vsdv8" podUID="f9a1488a-6bce-4e7b-846d-37825f27b619"
Sep 09 21:43:30.353995 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.353957    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7869998cc7-q2cj4"]
Sep 09 21:43:30.384505 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.384473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7869998cc7-q2cj4"]
Sep 09 21:43:30.384681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.384586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.387205 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.387181    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 21:43:30.387358 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.387336    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5mcm43ls77fms\""
Sep 09 21:43:30.387678 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.387660    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 21:43:30.388009 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.387986    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 21:43:30.388119 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.388027    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 21:43:30.388454 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.388437    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 21:43:30.388533 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.388464    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vcl7m\""
Sep 09 21:43:30.538016 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.537985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-tls\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.538201 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.538029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.538201 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.538108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzshh\" (UniqueName: \"kubernetes.io/projected/e7e916f3-911a-4310-b1a0-6bae8cfeee77-kube-api-access-nzshh\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.538201 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.538157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.538201 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.538179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.538201 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.538195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-grpc-tls\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.538379 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.538243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.538379 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.538302    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e916f3-911a-4310-b1a0-6bae8cfeee77-metrics-client-ca\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.639524 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.639437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.639524 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.639485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.639524 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.639513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-grpc-tls\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.640108 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.639549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.640108 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.639573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e916f3-911a-4310-b1a0-6bae8cfeee77-metrics-client-ca\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.640108 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.639611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-tls\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.640108 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.639665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.640108 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.639719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzshh\" (UniqueName: \"kubernetes.io/projected/e7e916f3-911a-4310-b1a0-6bae8cfeee77-kube-api-access-nzshh\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.640590 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.640534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e916f3-911a-4310-b1a0-6bae8cfeee77-metrics-client-ca\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.642494 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.642414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-tls\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.642494 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.642469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.642789 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.642747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.642887 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.642826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.642887 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.642862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-grpc-tls\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.642960 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.642923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7e916f3-911a-4310-b1a0-6bae8cfeee77-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.647874 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.647857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzshh\" (UniqueName: \"kubernetes.io/projected/e7e916f3-911a-4310-b1a0-6bae8cfeee77-kube-api-access-nzshh\") pod \"thanos-querier-7869998cc7-q2cj4\" (UID: \"e7e916f3-911a-4310-b1a0-6bae8cfeee77\") " pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:30.696180 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:30.696148    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:31.205091 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:31.205060    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7869998cc7-q2cj4"]
Sep 09 21:43:31.206823 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:43:31.206799    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e916f3_911a_4310_b1a0_6bae8cfeee77.slice/crio-3381d5034b4fba27d6c8138751e833e6ddd60124197ec0f199a8d4f40b673110 WatchSource:0}: Error finding container 3381d5034b4fba27d6c8138751e833e6ddd60124197ec0f199a8d4f40b673110: Status 404 returned error can't find the container with id 3381d5034b4fba27d6c8138751e833e6ddd60124197ec0f199a8d4f40b673110
Sep 09 21:43:31.395579 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:31.395548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:31.494970 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:31.494944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:31.600135 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:31.600087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:31.600321 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:31.600261    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7714c74-de4f-4360-a16a-5a68938045c6" containerID="d28f1f71756935aa26c5370b4109922532498a6e3cedf06ea69f9d75a27f9bc3" exitCode=0
Sep 09 21:43:31.605057 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:31.605030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghzwq" event={"ID":"f7714c74-de4f-4360-a16a-5a68938045c6","Type":"ContainerDied","Data":"d28f1f71756935aa26c5370b4109922532498a6e3cedf06ea69f9d75a27f9bc3"}
Sep 09 21:43:31.605162 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:31.605084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4" event={"ID":"e7e916f3-911a-4310-b1a0-6bae8cfeee77","Type":"ContainerStarted","Data":"3381d5034b4fba27d6c8138751e833e6ddd60124197ec0f199a8d4f40b673110"}
Sep 09 21:43:31.801875 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:31.801848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:32.025933 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:32.025854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:32.077922 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:32.077874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:32.608892 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:32.608854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghzwq" event={"ID":"f7714c74-de4f-4360-a16a-5a68938045c6","Type":"ContainerStarted","Data":"19af0de592bbbd2c802ae0e9611aa84bbfa8ad4e4b92ad65320a76f9b5216840"}
Sep 09 21:43:32.609099 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:32.608921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghzwq" event={"ID":"f7714c74-de4f-4360-a16a-5a68938045c6","Type":"ContainerStarted","Data":"98d7f8fd5971090d4cd4a58c0d1e072ce61824ef39736450f48a319950e5715c"}
Sep 09 21:43:32.628506 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:32.628455    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ghzwq" podStartSLOduration=3.022555268 podStartE2EDuration="4.62843925s" totalImagesPullingTime="1.518091846s" totalInitContainerRuntime="87.792136ms" isStatefulPod=true podCreationTimestamp="2026-09-09 21:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:32.626852756 +0000 UTC m=+305.523518770" watchObservedRunningTime="2026-09-09 21:43:32.62843925 +0000 UTC m=+305.525105257"
Sep 09 21:43:33.165301 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.163651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:33.167086 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.166953    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"]
Sep 09 21:43:33.190897 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.190847    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"]
Sep 09 21:43:33.191243 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.191224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"
Sep 09 21:43:33.194560 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.193640    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 21:43:33.194560 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.193945    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-98n8l\""
Sep 09 21:43:33.258259 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.258218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:33.305898 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.305855    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:33.306063 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.305937    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:33.307605 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.307578    2563 patch_prober.go:28] interesting pod/console-68df84699f-9jh42 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.19:8443/health\": dial tcp 10.132.0.19:8443: connect: connection refused" start-of-body=
Sep 09 21:43:33.307741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.307630    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-68df84699f-9jh42" podUID="f1b6c792-7dc5-42b8-b0e4-3d74481c150f" containerName="console" probeResult="failure" output="Get \"https://10.132.0.19:8443/health\": dial tcp 10.132.0.19:8443: connect: connection refused"
Sep 09 21:43:33.357851 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.357808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:33.362801 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.362753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60c67e91-3ce5-4a19-bb28-26afe4d49baa-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-mqp2s\" (UID: \"60c67e91-3ce5-4a19-bb28-26afe4d49baa\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"
Sep 09 21:43:33.362925 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.362866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rd6\" (UniqueName: \"kubernetes.io/projected/60c67e91-3ce5-4a19-bb28-26afe4d49baa-kube-api-access-k6rd6\") pod \"monitoring-plugin-85bbbbbf4-mqp2s\" (UID: \"60c67e91-3ce5-4a19-bb28-26afe4d49baa\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"
Sep 09 21:43:33.463503 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.463433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rd6\" (UniqueName: \"kubernetes.io/projected/60c67e91-3ce5-4a19-bb28-26afe4d49baa-kube-api-access-k6rd6\") pod \"monitoring-plugin-85bbbbbf4-mqp2s\" (UID: \"60c67e91-3ce5-4a19-bb28-26afe4d49baa\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"
Sep 09 21:43:33.463791 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.463741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60c67e91-3ce5-4a19-bb28-26afe4d49baa-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-mqp2s\" (UID: \"60c67e91-3ce5-4a19-bb28-26afe4d49baa\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"
Sep 09 21:43:33.466379 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.466356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60c67e91-3ce5-4a19-bb28-26afe4d49baa-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-mqp2s\" (UID: \"60c67e91-3ce5-4a19-bb28-26afe4d49baa\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"
Sep 09 21:43:33.471154 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.471129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rd6\" (UniqueName: \"kubernetes.io/projected/60c67e91-3ce5-4a19-bb28-26afe4d49baa-kube-api-access-k6rd6\") pod \"monitoring-plugin-85bbbbbf4-mqp2s\" (UID: \"60c67e91-3ce5-4a19-bb28-26afe4d49baa\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"
Sep 09 21:43:33.502938 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.502909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"
Sep 09 21:43:33.789460 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.789436    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"]
Sep 09 21:43:33.792387 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:43:33.792364    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c67e91_3ce5_4a19_bb28_26afe4d49baa.slice/crio-3246f4c7bc41ed21d657bc2d8d9437550dcb8d0d05b15e257b413fe1db43dae5 WatchSource:0}: Error finding container 3246f4c7bc41ed21d657bc2d8d9437550dcb8d0d05b15e257b413fe1db43dae5: Status 404 returned error can't find the container with id 3246f4c7bc41ed21d657bc2d8d9437550dcb8d0d05b15e257b413fe1db43dae5
Sep 09 21:43:33.933600 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:33.933537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:34.540432 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:34.540398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:34.574166 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:34.574135    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zh6lr"
Sep 09 21:43:34.620589 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:34.620549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4" event={"ID":"e7e916f3-911a-4310-b1a0-6bae8cfeee77","Type":"ContainerStarted","Data":"9ffc82c935341bda427e825c6bb3f84fea6a428e85f123745a01081989564892"}
Sep 09 21:43:34.620789 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:34.620648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4" event={"ID":"e7e916f3-911a-4310-b1a0-6bae8cfeee77","Type":"ContainerStarted","Data":"96462d7b75b8ae94590db6782ca51654b1c8afab3f8f499a809529754adaffbc"}
Sep 09 21:43:34.620789 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:34.620694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4" event={"ID":"e7e916f3-911a-4310-b1a0-6bae8cfeee77","Type":"ContainerStarted","Data":"a706580b7848c606a73e84b65646866048144434be60166b361e1587e8f42bc5"}
Sep 09 21:43:34.622068 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:34.622032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s" event={"ID":"60c67e91-3ce5-4a19-bb28-26afe4d49baa","Type":"ContainerStarted","Data":"3246f4c7bc41ed21d657bc2d8d9437550dcb8d0d05b15e257b413fe1db43dae5"}
Sep 09 21:43:34.760388 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:34.760351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:35.657873 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:35.657838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:36.634386 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:36.634304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4" event={"ID":"e7e916f3-911a-4310-b1a0-6bae8cfeee77","Type":"ContainerStarted","Data":"b57b505bd27bac4552e14957a682cdab1643e3a5ce8069730abdf495a0d1819d"}
Sep 09 21:43:36.634386 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:36.634379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4" event={"ID":"e7e916f3-911a-4310-b1a0-6bae8cfeee77","Type":"ContainerStarted","Data":"cb38c510abeaa3e17a14c405eaaf10b23af13fcbcdd1ff730574549ed3d02e27"}
Sep 09 21:43:36.634587 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:36.634433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4" event={"ID":"e7e916f3-911a-4310-b1a0-6bae8cfeee77","Type":"ContainerStarted","Data":"b62b7101264822f55711ac7bf89fe5493c2db744667de1dd8d5372954db5cbe2"}
Sep 09 21:43:36.634587 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:36.634479    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:36.635969 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:36.635946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s" event={"ID":"60c67e91-3ce5-4a19-bb28-26afe4d49baa","Type":"ContainerStarted","Data":"de0b9c54185c55680b0bd6a8ce909cc39f284c73f6fc67314a0d19c7e42381d7"}
Sep 09 21:43:36.636095 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:36.635999    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"
Sep 09 21:43:36.640754 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:36.640734    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s"
Sep 09 21:43:36.656679 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:36.656630    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4" podStartSLOduration=2.098901419 podStartE2EDuration="6.656614599s" totalImagesPullingTime="4.55771318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:36.654739208 +0000 UTC m=+309.551405241" watchObservedRunningTime="2026-09-09 21:43:36.656614599 +0000 UTC m=+309.553280608"
Sep 09 21:43:36.669803 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:36.669722    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-mqp2s" podStartSLOduration=1.370448732 podStartE2EDuration="3.669707199s" totalImagesPullingTime="2.299258467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:36.668325381 +0000 UTC m=+309.564991400" watchObservedRunningTime="2026-09-09 21:43:36.669707199 +0000 UTC m=+309.566373206"
Sep 09 21:43:39.215545 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:39.215504    2563 patch_prober.go:28] interesting pod/image-registry-b6d496f4d-s986w container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 21:43:39.215938 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:39.215557    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b6d496f4d-s986w" podUID="b9481e7b-af31-48ec-9504-a706c8d02cbd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:43:42.556179 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:42.556150    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:43:42.646461 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:42.646434    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7869998cc7-q2cj4"
Sep 09 21:43:42.809215 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:42.809129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:42.866451 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:42.866416    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:43.098071 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:43.097985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:43.146391 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:43.146350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:43.309583 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:43.309552    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:43.314009 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:43.313985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:43:43.449606 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:43.449485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:43.541900 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:43.541867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:44.104877 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:44.104842    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68df84699f-9jh42"]
Sep 09 21:43:44.197957 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:44.197916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:44.399406 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:44.399324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:44.458264 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:44.458230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:44.599943 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:44.599908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:43:48.277077 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:48.277042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:48.328011 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:48.327975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:50.824236 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:50.824203    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b6d496f4d-s986w"]
Sep 09 21:43:50.897904 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:50.897869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:50.998421 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:50.998391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:52.677750 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:52.677714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:52.728987 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:52.728955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:54.704065 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:54.703979    2563 generic.go:356] "Generic (PLEG): container finished" podID="ffe13b21-9884-46df-bcdc-169112e4b10d" containerID="6bbc05e6a00632b85f74c4e384361858e8b04cf49e3b2ea88b029699123c30b5" exitCode=0
Sep 09 21:43:54.704065 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:54.704052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x" event={"ID":"ffe13b21-9884-46df-bcdc-169112e4b10d","Type":"ContainerDied","Data":"6bbc05e6a00632b85f74c4e384361858e8b04cf49e3b2ea88b029699123c30b5"}
Sep 09 21:43:54.704543 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:54.704343    2563 scope.go:117] "RemoveContainer" containerID="6bbc05e6a00632b85f74c4e384361858e8b04cf49e3b2ea88b029699123c30b5"
Sep 09 21:43:55.710496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:55.710464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9gx7x" event={"ID":"ffe13b21-9884-46df-bcdc-169112e4b10d","Type":"ContainerStarted","Data":"1dd11b5d17cf356b9fba84c0139536aeb9bbe04b6fd574fd56abb8f76003d05a"}
Sep 09 21:43:56.662397 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:56.662359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:56.717292 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:43:56.717262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:02.054277 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:02.054235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:02.151662 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:02.151627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:05.208429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:05.208393    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n6cbw_0b8f9430-2b08-481b-9d13-4290193c0092/serve-healthcheck-canary/0.log"
Sep 09 21:44:05.409662 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:05.409630    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-hdssz_99f5bc95-24e1-4520-b21f-57edc4133ff8/cluster-samples-operator/0.log"
Sep 09 21:44:05.608480 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:05.608454    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-hdssz_99f5bc95-24e1-4520-b21f-57edc4133ff8/cluster-samples-operator-watch/0.log"
Sep 09 21:44:05.732159 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:05.732130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:05.787019 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:05.786990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:07.343849 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:07.343817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:07.404393 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:07.404364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:09.686430 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:09.686383    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68df84699f-9jh42" podUID="f1b6c792-7dc5-42b8-b0e4-3d74481c150f" containerName="console" containerID="cri-o://e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19" gracePeriod=15
Sep 09 21:44:09.929091 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:09.929064    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:44:10.063000 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.062966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-oauth-serving-cert\") pod \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") "
Sep 09 21:44:10.063000 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063004    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-trusted-ca-bundle\") pod \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") "
Sep 09 21:44:10.063259 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063056    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-service-ca\") pod \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") "
Sep 09 21:44:10.063259 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063085    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-oauth-config\") pod \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") "
Sep 09 21:44:10.063259 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063104    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-serving-cert\") pod \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") "
Sep 09 21:44:10.063259 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063128    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-config\") pod \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") "
Sep 09 21:44:10.063259 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063235    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whz5h\" (UniqueName: \"kubernetes.io/projected/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-kube-api-access-whz5h\") pod \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\" (UID: \"f1b6c792-7dc5-42b8-b0e4-3d74481c150f\") "
Sep 09 21:44:10.063506 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063404    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f1b6c792-7dc5-42b8-b0e4-3d74481c150f" (UID: "f1b6c792-7dc5-42b8-b0e4-3d74481c150f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:44:10.063564 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063537    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-oauth-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:10.063819 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063796    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f1b6c792-7dc5-42b8-b0e4-3d74481c150f" (UID: "f1b6c792-7dc5-42b8-b0e4-3d74481c150f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:44:10.063900 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-config" (OuterVolumeSpecName: "console-config") pod "f1b6c792-7dc5-42b8-b0e4-3d74481c150f" (UID: "f1b6c792-7dc5-42b8-b0e4-3d74481c150f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:44:10.063900 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.063823    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-service-ca" (OuterVolumeSpecName: "service-ca") pod "f1b6c792-7dc5-42b8-b0e4-3d74481c150f" (UID: "f1b6c792-7dc5-42b8-b0e4-3d74481c150f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:44:10.065950 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.065912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f1b6c792-7dc5-42b8-b0e4-3d74481c150f" (UID: "f1b6c792-7dc5-42b8-b0e4-3d74481c150f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:44:10.066057 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.065965    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-kube-api-access-whz5h" (OuterVolumeSpecName: "kube-api-access-whz5h") pod "f1b6c792-7dc5-42b8-b0e4-3d74481c150f" (UID: "f1b6c792-7dc5-42b8-b0e4-3d74481c150f"). InnerVolumeSpecName "kube-api-access-whz5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:44:10.066057 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.066012    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f1b6c792-7dc5-42b8-b0e4-3d74481c150f" (UID: "f1b6c792-7dc5-42b8-b0e4-3d74481c150f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:44:10.164219 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.164180    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-trusted-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:10.164219 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.164217    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:10.164431 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.164229    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-oauth-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:10.164431 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.164244    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:10.164431 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.164256    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-console-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:10.164431 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.164268    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whz5h\" (UniqueName: \"kubernetes.io/projected/f1b6c792-7dc5-42b8-b0e4-3d74481c150f-kube-api-access-whz5h\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:10.769581 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.769537    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1b6c792-7dc5-42b8-b0e4-3d74481c150f" containerID="e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19" exitCode=0
Sep 09 21:44:10.769581 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.769586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df84699f-9jh42" event={"ID":"f1b6c792-7dc5-42b8-b0e4-3d74481c150f","Type":"ContainerDied","Data":"e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19"}
Sep 09 21:44:10.770257 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.769598    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df84699f-9jh42"
Sep 09 21:44:10.770257 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.769625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df84699f-9jh42" event={"ID":"f1b6c792-7dc5-42b8-b0e4-3d74481c150f","Type":"ContainerDied","Data":"9de0abc3dc2b16be0b5d7e555dc01c1f515453b4636f619c59c0a92747a19802"}
Sep 09 21:44:10.770257 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.769645    2563 scope.go:117] "RemoveContainer" containerID="e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19"
Sep 09 21:44:10.783350 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.783328    2563 scope.go:117] "RemoveContainer" containerID="e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19"
Sep 09 21:44:10.783671 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:44:10.783652    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19\": container with ID starting with e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19 not found: ID does not exist" containerID="e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19"
Sep 09 21:44:10.783748 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.783676    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19"} err="failed to get container status \"e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19\": rpc error: code = NotFound desc = could not find container \"e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19\": container with ID starting with e6255a27349277bf61c5bbb31ba36a94f1064a77f669f3e0993de5594a06ff19 not found: ID does not exist"
Sep 09 21:44:10.795118 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.795092    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68df84699f-9jh42"]
Sep 09 21:44:10.798463 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:10.798442    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68df84699f-9jh42"]
Sep 09 21:44:11.604142 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:11.604109    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1b6c792-7dc5-42b8-b0e4-3d74481c150f" path="/var/lib/kubelet/pods/f1b6c792-7dc5-42b8-b0e4-3d74481c150f/volumes"
Sep 09 21:44:15.844976 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:15.844939    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b6d496f4d-s986w" podUID="b9481e7b-af31-48ec-9504-a706c8d02cbd" containerName="registry" containerID="cri-o://865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b" gracePeriod=30
Sep 09 21:44:16.099001 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.098931    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:44:16.216273 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.216237    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-certificates\") pod \"b9481e7b-af31-48ec-9504-a706c8d02cbd\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") "
Sep 09 21:44:16.216273 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.216275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9481e7b-af31-48ec-9504-a706c8d02cbd-ca-trust-extracted\") pod \"b9481e7b-af31-48ec-9504-a706c8d02cbd\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") "
Sep 09 21:44:16.216544 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.216304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-image-registry-private-configuration\") pod \"b9481e7b-af31-48ec-9504-a706c8d02cbd\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") "
Sep 09 21:44:16.216544 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.216326    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") pod \"b9481e7b-af31-48ec-9504-a706c8d02cbd\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") "
Sep 09 21:44:16.216544 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.216341    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-installation-pull-secrets\") pod \"b9481e7b-af31-48ec-9504-a706c8d02cbd\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") "
Sep 09 21:44:16.216544 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.216382    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vhtt\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-kube-api-access-6vhtt\") pod \"b9481e7b-af31-48ec-9504-a706c8d02cbd\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") "
Sep 09 21:44:16.216544 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.216421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-trusted-ca\") pod \"b9481e7b-af31-48ec-9504-a706c8d02cbd\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") "
Sep 09 21:44:16.216544 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.216450    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-bound-sa-token\") pod \"b9481e7b-af31-48ec-9504-a706c8d02cbd\" (UID: \"b9481e7b-af31-48ec-9504-a706c8d02cbd\") "
Sep 09 21:44:16.216922 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.216657    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b9481e7b-af31-48ec-9504-a706c8d02cbd" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:44:16.217681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.217632    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b9481e7b-af31-48ec-9504-a706c8d02cbd" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:44:16.219260 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.219203    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b9481e7b-af31-48ec-9504-a706c8d02cbd" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:44:16.219446 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.219422    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b9481e7b-af31-48ec-9504-a706c8d02cbd" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:44:16.219559 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.219471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b9481e7b-af31-48ec-9504-a706c8d02cbd" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:44:16.219792 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.219745    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b9481e7b-af31-48ec-9504-a706c8d02cbd" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:44:16.219987 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.219964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-kube-api-access-6vhtt" (OuterVolumeSpecName: "kube-api-access-6vhtt") pod "b9481e7b-af31-48ec-9504-a706c8d02cbd" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd"). InnerVolumeSpecName "kube-api-access-6vhtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:44:16.226336 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.226311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9481e7b-af31-48ec-9504-a706c8d02cbd-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b9481e7b-af31-48ec-9504-a706c8d02cbd" (UID: "b9481e7b-af31-48ec-9504-a706c8d02cbd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:44:16.316988 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.316955    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-certificates\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:16.316988 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.316984    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9481e7b-af31-48ec-9504-a706c8d02cbd-ca-trust-extracted\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:16.316988 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.316993    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-image-registry-private-configuration\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:16.317206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.317003    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-registry-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:16.317206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.317012    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9481e7b-af31-48ec-9504-a706c8d02cbd-installation-pull-secrets\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:16.317206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.317021    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vhtt\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-kube-api-access-6vhtt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:16.317206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.317029    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9481e7b-af31-48ec-9504-a706c8d02cbd-trusted-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:16.317206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.317037    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9481e7b-af31-48ec-9504-a706c8d02cbd-bound-sa-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:44:16.334285 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.334241    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:16.798042 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.797999    2563 generic.go:356] "Generic (PLEG): container finished" podID="b9481e7b-af31-48ec-9504-a706c8d02cbd" containerID="865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b" exitCode=0
Sep 09 21:44:16.798350 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.798326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b6d496f4d-s986w" event={"ID":"b9481e7b-af31-48ec-9504-a706c8d02cbd","Type":"ContainerDied","Data":"865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b"}
Sep 09 21:44:16.799820 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.798563    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6d496f4d-s986w"
Sep 09 21:44:16.799820 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.798642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b6d496f4d-s986w" event={"ID":"b9481e7b-af31-48ec-9504-a706c8d02cbd","Type":"ContainerDied","Data":"7e474bb553977a07f2b244c24722d90a0e1a61e3710fb0d64ecdbaeb5e6c9c36"}
Sep 09 21:44:16.799820 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.798680    2563 scope.go:117] "RemoveContainer" containerID="865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b"
Sep 09 21:44:16.809174 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.809151    2563 scope.go:117] "RemoveContainer" containerID="865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b"
Sep 09 21:44:16.809483 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:44:16.809464    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b\": container with ID starting with 865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b not found: ID does not exist" containerID="865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b"
Sep 09 21:44:16.809548 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.809491    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b"} err="failed to get container status \"865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b\": rpc error: code = NotFound desc = could not find container \"865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b\": container with ID starting with 865846a9cb423a56d6bf0ead77809d1db25f4fe9f2d53af2797363c622152f7b not found: ID does not exist"
Sep 09 21:44:16.821003 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.820973    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b6d496f4d-s986w"]
Sep 09 21:44:16.824423 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:16.824400    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b6d496f4d-s986w"]
Sep 09 21:44:17.605092 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:17.605065    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9481e7b-af31-48ec-9504-a706c8d02cbd" path="/var/lib/kubelet/pods/b9481e7b-af31-48ec-9504-a706c8d02cbd/volumes"
Sep 09 21:44:24.233043 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:24.233007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:24.333911 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:24.333873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:24.387230 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:24.387206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:28.241457 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:28.241426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:28.342628 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:28.342592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:32.021414 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:32.021381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:32.074979 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:32.074948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:40.511579 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:40.511530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:44:40.514051 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:40.514030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9a1488a-6bce-4e7b-846d-37825f27b619-metrics-certs\") pod \"network-metrics-daemon-vsdv8\" (UID: \"f9a1488a-6bce-4e7b-846d-37825f27b619\") " pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:44:40.703366 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:40.703336    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-znm4z\""
Sep 09 21:44:40.711218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:40.711200    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsdv8"
Sep 09 21:44:40.979423 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:40.979345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vsdv8"]
Sep 09 21:44:40.982198 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:44:40.982171    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a1488a_6bce_4e7b_846d_37825f27b619.slice/crio-678e9f06f89e833bd827ca797ec174605b27ab05f001bf3f6c9c4547a92cd182 WatchSource:0}: Error finding container 678e9f06f89e833bd827ca797ec174605b27ab05f001bf3f6c9c4547a92cd182: Status 404 returned error can't find the container with id 678e9f06f89e833bd827ca797ec174605b27ab05f001bf3f6c9c4547a92cd182
Sep 09 21:44:41.898272 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:41.898238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vsdv8" event={"ID":"f9a1488a-6bce-4e7b-846d-37825f27b619","Type":"ContainerStarted","Data":"678e9f06f89e833bd827ca797ec174605b27ab05f001bf3f6c9c4547a92cd182"}
Sep 09 21:44:42.904322 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:42.904290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vsdv8" event={"ID":"f9a1488a-6bce-4e7b-846d-37825f27b619","Type":"ContainerStarted","Data":"4fcf8a97bda26a8f9a0ec7b5874a55e2661dd6aaa6a3210ad3ba795db226d745"}
Sep 09 21:44:42.904697 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:42.904342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vsdv8" event={"ID":"f9a1488a-6bce-4e7b-846d-37825f27b619","Type":"ContainerStarted","Data":"cc6bbc040feaea9747098379aa38bfb9f0f23d9afdd4d0cd478420e7326cc3b7"}
Sep 09 21:44:42.920396 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:42.920349    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vsdv8" podStartSLOduration=374.756853901 podStartE2EDuration="6m15.920335544s" totalImagesPullingTime="1.163481643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:44:42.919050637 +0000 UTC m=+375.815716646" watchObservedRunningTime="2026-09-09 21:44:42.920335544 +0000 UTC m=+375.817001552"
Sep 09 21:44:43.015814 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:43.015785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:43.116756 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:43.116719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:47.033633 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:47.033596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:47.133708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:47.133675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:47.652783 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:47.652730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:47.865450 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:47.865410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:52.739721 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.739650    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"]
Sep 09 21:44:52.740167 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.740147    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1b6c792-7dc5-42b8-b0e4-3d74481c150f" containerName="console"
Sep 09 21:44:52.740217 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.740170    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1b6c792-7dc5-42b8-b0e4-3d74481c150f" containerName="console"
Sep 09 21:44:52.740217 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.740196    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9481e7b-af31-48ec-9504-a706c8d02cbd" containerName="registry"
Sep 09 21:44:52.740217 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.740203    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9481e7b-af31-48ec-9504-a706c8d02cbd" containerName="registry"
Sep 09 21:44:52.740327 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.740277    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1b6c792-7dc5-42b8-b0e4-3d74481c150f" containerName="console"
Sep 09 21:44:52.740327 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.740290    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9481e7b-af31-48ec-9504-a706c8d02cbd" containerName="registry"
Sep 09 21:44:52.742645 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.742618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.745025 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.744992    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 21:44:52.745299 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.745278    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 21:44:52.745424 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.745337    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 21:44:52.745701 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.745686    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 21:44:52.745803 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.745739    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 21:44:52.745869 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.745828    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kxmqc\""
Sep 09 21:44:52.752992 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.752503    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 21:44:52.753675 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.753653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"]
Sep 09 21:44:52.807706 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.807664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-federate-client-tls\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.807909 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.807713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-secret-telemeter-client\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.807909 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.807760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.807909 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.807816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.807909 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.807837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrlc\" (UniqueName: \"kubernetes.io/projected/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-kube-api-access-jmrlc\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.807909 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.807863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-serving-certs-ca-bundle\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.807909 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.807882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-telemeter-client-tls\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.808125 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.807959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-metrics-client-ca\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.908500 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.908469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-metrics-client-ca\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.908690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.908507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-federate-client-tls\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.908690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.908528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-secret-telemeter-client\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.908690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.908586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.908690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.908644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.908690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.908675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrlc\" (UniqueName: \"kubernetes.io/projected/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-kube-api-access-jmrlc\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.908983 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.908713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-serving-certs-ca-bundle\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.908983 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.908741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-telemeter-client-tls\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.909368 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.909317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-metrics-client-ca\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.909492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.909400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-serving-certs-ca-bundle\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.909636 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.909618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.911117 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.911098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-secret-telemeter-client\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.911678 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.911647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.911841 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.911827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-telemeter-client-tls\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.911956 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.911935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-federate-client-tls\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:52.917783 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:52.917744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrlc\" (UniqueName: \"kubernetes.io/projected/d8fa0a29-c6f4-48c3-8096-a67459be5ce4-kube-api-access-jmrlc\") pod \"telemeter-client-6989ff7f7d-dhzpt\" (UID: \"d8fa0a29-c6f4-48c3-8096-a67459be5ce4\") " pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:53.054212 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:53.054176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"
Sep 09 21:44:53.329737 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:53.329707    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt"]
Sep 09 21:44:53.331585 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:44:53.331558    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fa0a29_c6f4_48c3_8096_a67459be5ce4.slice/crio-440362e18f28719a580285df44a1be5f62df7473873e942907532e3a77a80779 WatchSource:0}: Error finding container 440362e18f28719a580285df44a1be5f62df7473873e942907532e3a77a80779: Status 404 returned error can't find the container with id 440362e18f28719a580285df44a1be5f62df7473873e942907532e3a77a80779
Sep 09 21:44:53.946639 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:53.946600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt" event={"ID":"d8fa0a29-c6f4-48c3-8096-a67459be5ce4","Type":"ContainerStarted","Data":"440362e18f28719a580285df44a1be5f62df7473873e942907532e3a77a80779"}
Sep 09 21:44:55.958430 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:55.958392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt" event={"ID":"d8fa0a29-c6f4-48c3-8096-a67459be5ce4","Type":"ContainerStarted","Data":"738c7ca77d29c7d53b49239c8a91afcb954705cceba5be5ef7fa57c9c33dfe37"}
Sep 09 21:44:56.963812 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:56.963757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt" event={"ID":"d8fa0a29-c6f4-48c3-8096-a67459be5ce4","Type":"ContainerStarted","Data":"6ea262eec891bb5c58c793bb7cbce091811bc4396cb763c43dbd87e8b0bb0a76"}
Sep 09 21:44:56.964188 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:56.963833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt" event={"ID":"d8fa0a29-c6f4-48c3-8096-a67459be5ce4","Type":"ContainerStarted","Data":"f5101fe67bf5c8562eac7a87a6adce859739f12fe4292292eb3aa929a5448be3"}
Sep 09 21:44:56.995338 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:56.995282    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6989ff7f7d-dhzpt" podStartSLOduration=1.897408572 podStartE2EDuration="4.995262623s" totalImagesPullingTime="3.097854051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:44:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:44:56.992649984 +0000 UTC m=+389.889315999" watchObservedRunningTime="2026-09-09 21:44:56.995262623 +0000 UTC m=+389.891928632"
Sep 09 21:44:57.066938 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.066906    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f49f94586-zc4cw"]
Sep 09 21:44:57.069666 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.069638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.072134 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.072114    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 21:44:57.072288 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.072269    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 21:44:57.072372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.072287    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 21:44:57.072372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.072269    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hzl8x\""
Sep 09 21:44:57.072372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.072269    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 21:44:57.072557 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.072540    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 21:44:57.081966 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.081906    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 21:44:57.082588 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.082568    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f49f94586-zc4cw"]
Sep 09 21:44:57.127811 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.127759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:57.139984 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.139950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-trusted-ca-bundle\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.139984 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.139982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xqz\" (UniqueName: \"kubernetes.io/projected/2a14800c-cb0a-4b65-8c97-9871f4086781-kube-api-access-f6xqz\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.140181 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.140000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-oauth-serving-cert\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.140181 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.140053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-serving-cert\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.140181 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.140125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-oauth-config\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.140181 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.140148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-console-config\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.140318 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.140189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-service-ca\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.220154 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.220061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:57.241170 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.241137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-trusted-ca-bundle\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.241170 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.241170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xqz\" (UniqueName: \"kubernetes.io/projected/2a14800c-cb0a-4b65-8c97-9871f4086781-kube-api-access-f6xqz\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.241358 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.241188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-oauth-serving-cert\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.241358 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.241240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-serving-cert\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.241358 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.241307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-oauth-config\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.241358 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.241334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-console-config\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.241547 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.241375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-service-ca\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.242069 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.242046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-oauth-serving-cert\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.242186 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.242133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-service-ca\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.242186 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.242157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-trusted-ca-bundle\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.242347 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.242324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-console-config\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.243874 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.243850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-oauth-config\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.243971 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.243933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-serving-cert\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.248418 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.248397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xqz\" (UniqueName: \"kubernetes.io/projected/2a14800c-cb0a-4b65-8c97-9871f4086781-kube-api-access-f6xqz\") pod \"console-6f49f94586-zc4cw\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") " pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.321612 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.321568    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:57.383121 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.383087    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:44:57.651345 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.651317    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f49f94586-zc4cw"]
Sep 09 21:44:57.654684 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:44:57.654658    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a14800c_cb0a_4b65_8c97_9871f4086781.slice/crio-ca62050c811b178beeda02c85c5dc57be46e0e414762d5b1b75b9d4365f2d27d WatchSource:0}: Error finding container ca62050c811b178beeda02c85c5dc57be46e0e414762d5b1b75b9d4365f2d27d: Status 404 returned error can't find the container with id ca62050c811b178beeda02c85c5dc57be46e0e414762d5b1b75b9d4365f2d27d
Sep 09 21:44:57.881488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.881457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:57.970110 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.970012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f49f94586-zc4cw" event={"ID":"2a14800c-cb0a-4b65-8c97-9871f4086781","Type":"ContainerStarted","Data":"68aa1aca3b2c27ef12d93214e4f038948080d9413fe8db72212a836bf8c1a6ea"}
Sep 09 21:44:57.970110 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.970066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f49f94586-zc4cw" event={"ID":"2a14800c-cb0a-4b65-8c97-9871f4086781","Type":"ContainerStarted","Data":"ca62050c811b178beeda02c85c5dc57be46e0e414762d5b1b75b9d4365f2d27d"}
Sep 09 21:44:57.982169 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:44:57.982134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:02.882468 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:02.882424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:02.945988 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:02.945960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:05.932139 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:05.932071    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f49f94586-zc4cw" podStartSLOduration=8.932051893 podStartE2EDuration="8.932051893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:44:57.988919676 +0000 UTC m=+390.885585703" watchObservedRunningTime="2026-09-09 21:45:05.932051893 +0000 UTC m=+398.828717902"
Sep 09 21:45:05.932645 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:05.932629    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f49f94586-zc4cw"]
Sep 09 21:45:05.963221 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:05.963187    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f4c554954-6jsl5"]
Sep 09 21:45:05.965464 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:05.965446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:05.978675 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:05.978646    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f4c554954-6jsl5"]
Sep 09 21:45:06.012174 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.012128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-serving-cert\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.012174 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.012176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-config\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.012417 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.012218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9qb\" (UniqueName: \"kubernetes.io/projected/f55e5f7e-f2cd-45e7-bb97-866da2074892-kube-api-access-mb9qb\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.012417 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.012265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-oauth-config\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.012417 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.012294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-trusted-ca-bundle\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.012417 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.012324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-oauth-serving-cert\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.012417 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.012392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-service-ca\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.112972 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.112935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-service-ca\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.112972 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.112980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-serving-cert\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.113242 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.112996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-config\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.113242 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.113022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9qb\" (UniqueName: \"kubernetes.io/projected/f55e5f7e-f2cd-45e7-bb97-866da2074892-kube-api-access-mb9qb\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.113242 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.113161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-oauth-config\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.113242 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.113201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-trusted-ca-bundle\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.113242 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.113235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-oauth-serving-cert\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.113762 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.113735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-service-ca\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.113914 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.113870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-config\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.114042 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.114022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-trusted-ca-bundle\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.114208 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.114186    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-oauth-serving-cert\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.115708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.115678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-oauth-config\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.116071 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.116052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-serving-cert\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.121588 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.121554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9qb\" (UniqueName: \"kubernetes.io/projected/f55e5f7e-f2cd-45e7-bb97-866da2074892-kube-api-access-mb9qb\") pod \"console-f4c554954-6jsl5\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") " pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.126105 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.126080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:06.274475 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.274442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:06.546209 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.546183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f4c554954-6jsl5"]
Sep 09 21:45:06.548834 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:45:06.548804    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55e5f7e_f2cd_45e7_bb97_866da2074892.slice/crio-51d0fb0592d45060b2a7919c4594712ea9f6b3c764b59c727690340c0c66c09c WatchSource:0}: Error finding container 51d0fb0592d45060b2a7919c4594712ea9f6b3c764b59c727690340c0c66c09c: Status 404 returned error can't find the container with id 51d0fb0592d45060b2a7919c4594712ea9f6b3c764b59c727690340c0c66c09c
Sep 09 21:45:06.723933 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.723904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:06.777837 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:06.777807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:07.006041 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:07.006005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4c554954-6jsl5" event={"ID":"f55e5f7e-f2cd-45e7-bb97-866da2074892","Type":"ContainerStarted","Data":"707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51"}
Sep 09 21:45:07.006425 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:07.006055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4c554954-6jsl5" event={"ID":"f55e5f7e-f2cd-45e7-bb97-866da2074892","Type":"ContainerStarted","Data":"51d0fb0592d45060b2a7919c4594712ea9f6b3c764b59c727690340c0c66c09c"}
Sep 09 21:45:07.024165 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:07.024117    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f4c554954-6jsl5" podStartSLOduration=2.024097572 podStartE2EDuration="2.024097572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:07.02209493 +0000 UTC m=+399.918760939" watchObservedRunningTime="2026-09-09 21:45:07.024097572 +0000 UTC m=+399.920763583"
Sep 09 21:45:07.383832 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:07.383742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:45:14.749941 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:14.749902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:14.851970 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:14.851936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:16.274626 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:16.274583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:16.275077 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:16.274671    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:16.279570 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:16.279540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:17.049052 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:17.049026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:45:17.187645 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:17.187608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:17.237358 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:17.237333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:17.298328 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:17.298290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:17.350702 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:17.350624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:30.953169 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:30.953128    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f49f94586-zc4cw" podUID="2a14800c-cb0a-4b65-8c97-9871f4086781" containerName="console" containerID="cri-o://68aa1aca3b2c27ef12d93214e4f038948080d9413fe8db72212a836bf8c1a6ea" gracePeriod=15
Sep 09 21:45:31.101058 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.101026    2563 generic.go:356] "Generic (PLEG): container finished" podID="2a14800c-cb0a-4b65-8c97-9871f4086781" containerID="68aa1aca3b2c27ef12d93214e4f038948080d9413fe8db72212a836bf8c1a6ea" exitCode=0
Sep 09 21:45:31.101215 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.101095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f49f94586-zc4cw" event={"ID":"2a14800c-cb0a-4b65-8c97-9871f4086781","Type":"ContainerDied","Data":"68aa1aca3b2c27ef12d93214e4f038948080d9413fe8db72212a836bf8c1a6ea"}
Sep 09 21:45:31.203321 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.203265    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:45:31.323005 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.322967    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-oauth-config\") pod \"2a14800c-cb0a-4b65-8c97-9871f4086781\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") "
Sep 09 21:45:31.323005 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.323011    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-trusted-ca-bundle\") pod \"2a14800c-cb0a-4b65-8c97-9871f4086781\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") "
Sep 09 21:45:31.323250 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.323028    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-serving-cert\") pod \"2a14800c-cb0a-4b65-8c97-9871f4086781\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") "
Sep 09 21:45:31.323250 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.323076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-service-ca\") pod \"2a14800c-cb0a-4b65-8c97-9871f4086781\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") "
Sep 09 21:45:31.323250 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.323099    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-oauth-serving-cert\") pod \"2a14800c-cb0a-4b65-8c97-9871f4086781\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") "
Sep 09 21:45:31.323250 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.323114    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-console-config\") pod \"2a14800c-cb0a-4b65-8c97-9871f4086781\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") "
Sep 09 21:45:31.323250 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.323171    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6xqz\" (UniqueName: \"kubernetes.io/projected/2a14800c-cb0a-4b65-8c97-9871f4086781-kube-api-access-f6xqz\") pod \"2a14800c-cb0a-4b65-8c97-9871f4086781\" (UID: \"2a14800c-cb0a-4b65-8c97-9871f4086781\") "
Sep 09 21:45:31.323552 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.323521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2a14800c-cb0a-4b65-8c97-9871f4086781" (UID: "2a14800c-cb0a-4b65-8c97-9871f4086781"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:45:31.323552 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.323530    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-service-ca" (OuterVolumeSpecName: "service-ca") pod "2a14800c-cb0a-4b65-8c97-9871f4086781" (UID: "2a14800c-cb0a-4b65-8c97-9871f4086781"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:45:31.323682 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.323539    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2a14800c-cb0a-4b65-8c97-9871f4086781" (UID: "2a14800c-cb0a-4b65-8c97-9871f4086781"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:45:31.323682 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.323550    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-console-config" (OuterVolumeSpecName: "console-config") pod "2a14800c-cb0a-4b65-8c97-9871f4086781" (UID: "2a14800c-cb0a-4b65-8c97-9871f4086781"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:45:31.325371 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.325346    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2a14800c-cb0a-4b65-8c97-9871f4086781" (UID: "2a14800c-cb0a-4b65-8c97-9871f4086781"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:45:31.325371 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.325357    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2a14800c-cb0a-4b65-8c97-9871f4086781" (UID: "2a14800c-cb0a-4b65-8c97-9871f4086781"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:45:31.325528 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.325401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a14800c-cb0a-4b65-8c97-9871f4086781-kube-api-access-f6xqz" (OuterVolumeSpecName: "kube-api-access-f6xqz") pod "2a14800c-cb0a-4b65-8c97-9871f4086781" (UID: "2a14800c-cb0a-4b65-8c97-9871f4086781"). InnerVolumeSpecName "kube-api-access-f6xqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:45:31.424106 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.424059    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6xqz\" (UniqueName: \"kubernetes.io/projected/2a14800c-cb0a-4b65-8c97-9871f4086781-kube-api-access-f6xqz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:45:31.424106 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.424102    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-oauth-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:45:31.424106 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.424112    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-trusted-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:45:31.424106 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.424121    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a14800c-cb0a-4b65-8c97-9871f4086781-console-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:45:31.424374 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.424130    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:45:31.424374 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.424138    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-oauth-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:45:31.424374 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:31.424147    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a14800c-cb0a-4b65-8c97-9871f4086781-console-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:45:32.043415 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:32.043377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:32.106354 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:32.106325    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f49f94586-zc4cw"
Sep 09 21:45:32.106528 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:32.106324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f49f94586-zc4cw" event={"ID":"2a14800c-cb0a-4b65-8c97-9871f4086781","Type":"ContainerDied","Data":"ca62050c811b178beeda02c85c5dc57be46e0e414762d5b1b75b9d4365f2d27d"}
Sep 09 21:45:32.106528 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:32.106451    2563 scope.go:117] "RemoveContainer" containerID="68aa1aca3b2c27ef12d93214e4f038948080d9413fe8db72212a836bf8c1a6ea"
Sep 09 21:45:32.129248 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:32.129221    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f49f94586-zc4cw"]
Sep 09 21:45:32.134613 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:32.134586    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f49f94586-zc4cw"]
Sep 09 21:45:32.136454 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:32.136421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:33.604672 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:33.604636    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a14800c-cb0a-4b65-8c97-9871f4086781" path="/var/lib/kubelet/pods/2a14800c-cb0a-4b65-8c97-9871f4086781/volumes"
Sep 09 21:45:34.974595 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:34.974563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:35.024680 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:45:35.024629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:09.262576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:09.262466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:20.311143 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:20.311095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:20.391746 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:20.391713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:33.643803 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:33.643727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:33.744873 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:33.744790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:34.345802 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.345754    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"]
Sep 09 21:46:34.346101 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.346089    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a14800c-cb0a-4b65-8c97-9871f4086781" containerName="console"
Sep 09 21:46:34.346143 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.346103    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a14800c-cb0a-4b65-8c97-9871f4086781" containerName="console"
Sep 09 21:46:34.346177 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.346163    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a14800c-cb0a-4b65-8c97-9871f4086781" containerName="console"
Sep 09 21:46:34.349548 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.349531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.352368 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.352341    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 21:46:34.352495 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.352393    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nnpqj\""
Sep 09 21:46:34.352495 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.352412    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 21:46:34.356458 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.356233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"]
Sep 09 21:46:34.436916 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.436874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.437088 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.436928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.437088 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.436953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvhqh\" (UniqueName: \"kubernetes.io/projected/673da5f3-1211-43d5-8ca3-a35db4ffcfac-kube-api-access-cvhqh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.538047 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.538010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.538243 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.538061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.538243 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.538099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvhqh\" (UniqueName: \"kubernetes.io/projected/673da5f3-1211-43d5-8ca3-a35db4ffcfac-kube-api-access-cvhqh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.538467 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.538445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.538467 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.538460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.546420 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.546391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvhqh\" (UniqueName: \"kubernetes.io/projected/673da5f3-1211-43d5-8ca3-a35db4ffcfac-kube-api-access-cvhqh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.659610 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.659530    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:34.929568 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:34.929519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"]
Sep 09 21:46:34.931722 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:46:34.931694    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673da5f3_1211_43d5_8ca3_a35db4ffcfac.slice/crio-54c44db86a1919d84abbb3dbca6fb777d7f9c5f019e5e65f25033f1d5d62d66d WatchSource:0}: Error finding container 54c44db86a1919d84abbb3dbca6fb777d7f9c5f019e5e65f25033f1d5d62d66d: Status 404 returned error can't find the container with id 54c44db86a1919d84abbb3dbca6fb777d7f9c5f019e5e65f25033f1d5d62d66d
Sep 09 21:46:35.357871 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:35.357839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm" event={"ID":"673da5f3-1211-43d5-8ca3-a35db4ffcfac","Type":"ContainerStarted","Data":"54c44db86a1919d84abbb3dbca6fb777d7f9c5f019e5e65f25033f1d5d62d66d"}
Sep 09 21:46:42.390400 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:42.390361    2563 generic.go:356] "Generic (PLEG): container finished" podID="673da5f3-1211-43d5-8ca3-a35db4ffcfac" containerID="e42561e4b3a66c643772864fbdb556fd60486e932ca329a175caef42ec9b239d" exitCode=0
Sep 09 21:46:42.390819 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:42.390452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm" event={"ID":"673da5f3-1211-43d5-8ca3-a35db4ffcfac","Type":"ContainerDied","Data":"e42561e4b3a66c643772864fbdb556fd60486e932ca329a175caef42ec9b239d"}
Sep 09 21:46:45.404820 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:45.404785    2563 generic.go:356] "Generic (PLEG): container finished" podID="673da5f3-1211-43d5-8ca3-a35db4ffcfac" containerID="e255e0a04d50fe31722aa37f2db6273c4c3892dc9786e781f1bda3a18918605b" exitCode=0
Sep 09 21:46:45.405187 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:45.404862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm" event={"ID":"673da5f3-1211-43d5-8ca3-a35db4ffcfac","Type":"ContainerDied","Data":"e255e0a04d50fe31722aa37f2db6273c4c3892dc9786e781f1bda3a18918605b"}
Sep 09 21:46:52.439954 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:52.439912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm" event={"ID":"673da5f3-1211-43d5-8ca3-a35db4ffcfac","Type":"ContainerStarted","Data":"f70297fc07e88d9f2f717d9ac54b5d6fb6afae2253689cc3603cd689fe5de9ff"}
Sep 09 21:46:52.459349 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:52.459301    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm" podStartSLOduration=2.759074025 podStartE2EDuration="18.459286799s" totalImagesPullingTime="15.700212774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:52.457969004 +0000 UTC m=+505.354635011" watchObservedRunningTime="2026-09-09 21:46:52.459286799 +0000 UTC m=+505.355952806"
Sep 09 21:46:53.446651 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:53.446614    2563 generic.go:356] "Generic (PLEG): container finished" podID="673da5f3-1211-43d5-8ca3-a35db4ffcfac" containerID="f70297fc07e88d9f2f717d9ac54b5d6fb6afae2253689cc3603cd689fe5de9ff" exitCode=0
Sep 09 21:46:53.447077 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:53.446740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm" event={"ID":"673da5f3-1211-43d5-8ca3-a35db4ffcfac","Type":"ContainerDied","Data":"f70297fc07e88d9f2f717d9ac54b5d6fb6afae2253689cc3603cd689fe5de9ff"}
Sep 09 21:46:54.627363 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:54.627325    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:54.702020 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:54.701938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-bundle\") pod \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") "
Sep 09 21:46:54.702020 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:54.702000    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvhqh\" (UniqueName: \"kubernetes.io/projected/673da5f3-1211-43d5-8ca3-a35db4ffcfac-kube-api-access-cvhqh\") pod \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") "
Sep 09 21:46:54.702230 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:54.702027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-util\") pod \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\" (UID: \"673da5f3-1211-43d5-8ca3-a35db4ffcfac\") "
Sep 09 21:46:54.702572 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:54.702546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-bundle" (OuterVolumeSpecName: "bundle") pod "673da5f3-1211-43d5-8ca3-a35db4ffcfac" (UID: "673da5f3-1211-43d5-8ca3-a35db4ffcfac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:46:54.704300 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:54.704277    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673da5f3-1211-43d5-8ca3-a35db4ffcfac-kube-api-access-cvhqh" (OuterVolumeSpecName: "kube-api-access-cvhqh") pod "673da5f3-1211-43d5-8ca3-a35db4ffcfac" (UID: "673da5f3-1211-43d5-8ca3-a35db4ffcfac"). InnerVolumeSpecName "kube-api-access-cvhqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:46:54.707714 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:54.707689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-util" (OuterVolumeSpecName: "util") pod "673da5f3-1211-43d5-8ca3-a35db4ffcfac" (UID: "673da5f3-1211-43d5-8ca3-a35db4ffcfac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:46:54.803488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:54.803447    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:46:54.803488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:54.803479    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvhqh\" (UniqueName: \"kubernetes.io/projected/673da5f3-1211-43d5-8ca3-a35db4ffcfac-kube-api-access-cvhqh\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:46:54.803488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:54.803491    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/673da5f3-1211-43d5-8ca3-a35db4ffcfac-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:46:55.456756 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:55.456730    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm"
Sep 09 21:46:55.456756 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:55.456741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wslpm" event={"ID":"673da5f3-1211-43d5-8ca3-a35db4ffcfac","Type":"ContainerDied","Data":"54c44db86a1919d84abbb3dbca6fb777d7f9c5f019e5e65f25033f1d5d62d66d"}
Sep 09 21:46:55.456962 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:46:55.456798    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54c44db86a1919d84abbb3dbca6fb777d7f9c5f019e5e65f25033f1d5d62d66d"
Sep 09 21:47:00.187130 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:00.187087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:01.603953 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:01.603921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:02.150702 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.150668    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"]
Sep 09 21:47:02.151011 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.150999    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673da5f3-1211-43d5-8ca3-a35db4ffcfac" containerName="util"
Sep 09 21:47:02.151056 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.151013    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="673da5f3-1211-43d5-8ca3-a35db4ffcfac" containerName="util"
Sep 09 21:47:02.151056 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.151020    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673da5f3-1211-43d5-8ca3-a35db4ffcfac" containerName="extract"
Sep 09 21:47:02.151056 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.151025    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="673da5f3-1211-43d5-8ca3-a35db4ffcfac" containerName="extract"
Sep 09 21:47:02.151056 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.151043    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673da5f3-1211-43d5-8ca3-a35db4ffcfac" containerName="pull"
Sep 09 21:47:02.151056 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.151049    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="673da5f3-1211-43d5-8ca3-a35db4ffcfac" containerName="pull"
Sep 09 21:47:02.151199 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.151099    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="673da5f3-1211-43d5-8ca3-a35db4ffcfac" containerName="extract"
Sep 09 21:47:02.152936 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.152921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"
Sep 09 21:47:02.156396 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.156374    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kc9fz\""
Sep 09 21:47:02.156527 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.156509    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 21:47:02.157335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.157322    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:47:02.172629 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.172594    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"]
Sep 09 21:47:02.259247 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.259203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6wg\" (UniqueName: \"kubernetes.io/projected/1561f846-a374-47bb-98f6-13ffb2997ee3-kube-api-access-xc6wg\") pod \"cert-manager-operator-controller-manager-55c444bf86-vhtjt\" (UID: \"1561f846-a374-47bb-98f6-13ffb2997ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"
Sep 09 21:47:02.259427 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.259263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1561f846-a374-47bb-98f6-13ffb2997ee3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vhtjt\" (UID: \"1561f846-a374-47bb-98f6-13ffb2997ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"
Sep 09 21:47:02.360538 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.360500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6wg\" (UniqueName: \"kubernetes.io/projected/1561f846-a374-47bb-98f6-13ffb2997ee3-kube-api-access-xc6wg\") pod \"cert-manager-operator-controller-manager-55c444bf86-vhtjt\" (UID: \"1561f846-a374-47bb-98f6-13ffb2997ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"
Sep 09 21:47:02.360690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.360565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1561f846-a374-47bb-98f6-13ffb2997ee3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vhtjt\" (UID: \"1561f846-a374-47bb-98f6-13ffb2997ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"
Sep 09 21:47:02.360985 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.360964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1561f846-a374-47bb-98f6-13ffb2997ee3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vhtjt\" (UID: \"1561f846-a374-47bb-98f6-13ffb2997ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"
Sep 09 21:47:02.370284 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.370258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6wg\" (UniqueName: \"kubernetes.io/projected/1561f846-a374-47bb-98f6-13ffb2997ee3-kube-api-access-xc6wg\") pod \"cert-manager-operator-controller-manager-55c444bf86-vhtjt\" (UID: \"1561f846-a374-47bb-98f6-13ffb2997ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"
Sep 09 21:47:02.461581 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.461493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"
Sep 09 21:47:02.744147 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:02.744120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt"]
Sep 09 21:47:02.746682 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:47:02.746652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1561f846_a374_47bb_98f6_13ffb2997ee3.slice/crio-76f3477c208b0da5c516cc6cfd539e420d02e6698a52da58b155e730d5ec5c63 WatchSource:0}: Error finding container 76f3477c208b0da5c516cc6cfd539e420d02e6698a52da58b155e730d5ec5c63: Status 404 returned error can't find the container with id 76f3477c208b0da5c516cc6cfd539e420d02e6698a52da58b155e730d5ec5c63
Sep 09 21:47:03.493323 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:03.493279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt" event={"ID":"1561f846-a374-47bb-98f6-13ffb2997ee3","Type":"ContainerStarted","Data":"76f3477c208b0da5c516cc6cfd539e420d02e6698a52da58b155e730d5ec5c63"}
Sep 09 21:47:04.054115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:04.054083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:04.178904 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:04.178859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:04.500137 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:04.500095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt" event={"ID":"1561f846-a374-47bb-98f6-13ffb2997ee3","Type":"ContainerStarted","Data":"33642951fd6bdecc9d0d441ad24716219b734cd481a5cb4da545e995a8e8593c"}
Sep 09 21:47:04.534681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:04.534615    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vhtjt" podStartSLOduration=0.924212641 podStartE2EDuration="2.534595488s" totalImagesPullingTime="1.610382847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:04.533999976 +0000 UTC m=+517.430665985" watchObservedRunningTime="2026-09-09 21:47:04.534595488 +0000 UTC m=+517.431261495"
Sep 09 21:47:04.991650 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:04.991603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:05.282059 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:05.282027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:05.961413 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:05.961379    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"]
Sep 09 21:47:05.963823 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:05.963806    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:05.966916 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:05.966894    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 21:47:05.967030 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:05.966898    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 21:47:05.967856 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:05.967838    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nnpqj\""
Sep 09 21:47:05.977937 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:05.977914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"]
Sep 09 21:47:05.991505 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:05.991477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:05.991630 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:05.991526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:05.991630 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:05.991548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9m5\" (UniqueName: \"kubernetes.io/projected/cf19da70-632f-415d-9a12-9c528cbfe09f-kube-api-access-dm9m5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:06.093008 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.092963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:06.093204 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.093030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:06.093204 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.093057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9m5\" (UniqueName: \"kubernetes.io/projected/cf19da70-632f-415d-9a12-9c528cbfe09f-kube-api-access-dm9m5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:06.093384 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.093363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:06.093384 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.093374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:06.105888 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.105862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9m5\" (UniqueName: \"kubernetes.io/projected/cf19da70-632f-415d-9a12-9c528cbfe09f-kube-api-access-dm9m5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:06.273054 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.273026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:06.344205 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.344176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:06.437241 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.437203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:06.544515 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.544407    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"]
Sep 09 21:47:06.547157 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:47:06.547131    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf19da70_632f_415d_9a12_9c528cbfe09f.slice/crio-c3e6d14e1c8f253308d2857752a24da87805095fc2a0a8003c2197f918aed3de WatchSource:0}: Error finding container c3e6d14e1c8f253308d2857752a24da87805095fc2a0a8003c2197f918aed3de: Status 404 returned error can't find the container with id c3e6d14e1c8f253308d2857752a24da87805095fc2a0a8003c2197f918aed3de
Sep 09 21:47:06.848150 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.848068    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-q8tlz"]
Sep 09 21:47:06.850326 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.850307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz"
Sep 09 21:47:06.853055 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.853035    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-76djb\""
Sep 09 21:47:06.853055 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.853047    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 21:47:06.854089 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.854073    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 21:47:06.863862 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.863838    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-q8tlz"]
Sep 09 21:47:06.898685 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.898654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvgq\" (UniqueName: \"kubernetes.io/projected/ebe58885-2cc8-4d34-9ec8-86f74e88ffe1-kube-api-access-cbvgq\") pod \"cert-manager-webhook-6486ffd997-q8tlz\" (UID: \"ebe58885-2cc8-4d34-9ec8-86f74e88ffe1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz"
Sep 09 21:47:06.898685 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.898688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebe58885-2cc8-4d34-9ec8-86f74e88ffe1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-q8tlz\" (UID: \"ebe58885-2cc8-4d34-9ec8-86f74e88ffe1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz"
Sep 09 21:47:06.999383 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.999346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvgq\" (UniqueName: \"kubernetes.io/projected/ebe58885-2cc8-4d34-9ec8-86f74e88ffe1-kube-api-access-cbvgq\") pod \"cert-manager-webhook-6486ffd997-q8tlz\" (UID: \"ebe58885-2cc8-4d34-9ec8-86f74e88ffe1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz"
Sep 09 21:47:06.999383 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:06.999383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebe58885-2cc8-4d34-9ec8-86f74e88ffe1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-q8tlz\" (UID: \"ebe58885-2cc8-4d34-9ec8-86f74e88ffe1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz"
Sep 09 21:47:07.012977 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:07.012944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebe58885-2cc8-4d34-9ec8-86f74e88ffe1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-q8tlz\" (UID: \"ebe58885-2cc8-4d34-9ec8-86f74e88ffe1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz"
Sep 09 21:47:07.013094 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:07.013009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvgq\" (UniqueName: \"kubernetes.io/projected/ebe58885-2cc8-4d34-9ec8-86f74e88ffe1-kube-api-access-cbvgq\") pod \"cert-manager-webhook-6486ffd997-q8tlz\" (UID: \"ebe58885-2cc8-4d34-9ec8-86f74e88ffe1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz"
Sep 09 21:47:07.173708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:07.173673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz"
Sep 09 21:47:07.447588 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:07.447500    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-q8tlz"]
Sep 09 21:47:07.449801 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:47:07.449756    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebe58885_2cc8_4d34_9ec8_86f74e88ffe1.slice/crio-1edb90f0b1fbe8d3b65fadd225e9c4ac554bd8d0e975d6abeb55e6ed1a276293 WatchSource:0}: Error finding container 1edb90f0b1fbe8d3b65fadd225e9c4ac554bd8d0e975d6abeb55e6ed1a276293: Status 404 returned error can't find the container with id 1edb90f0b1fbe8d3b65fadd225e9c4ac554bd8d0e975d6abeb55e6ed1a276293
Sep 09 21:47:07.516147 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:07.516111    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf19da70-632f-415d-9a12-9c528cbfe09f" containerID="73bcb1ba21a8d4b4ce458995b079d3c7c7388792e67d34e473693abf4b2e9855" exitCode=0
Sep 09 21:47:07.516331 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:07.516193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548" event={"ID":"cf19da70-632f-415d-9a12-9c528cbfe09f","Type":"ContainerDied","Data":"73bcb1ba21a8d4b4ce458995b079d3c7c7388792e67d34e473693abf4b2e9855"}
Sep 09 21:47:07.516331 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:07.516245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548" event={"ID":"cf19da70-632f-415d-9a12-9c528cbfe09f","Type":"ContainerStarted","Data":"c3e6d14e1c8f253308d2857752a24da87805095fc2a0a8003c2197f918aed3de"}
Sep 09 21:47:07.517500 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:07.517473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz" event={"ID":"ebe58885-2cc8-4d34-9ec8-86f74e88ffe1","Type":"ContainerStarted","Data":"1edb90f0b1fbe8d3b65fadd225e9c4ac554bd8d0e975d6abeb55e6ed1a276293"}
Sep 09 21:47:08.540828 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:08.540792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:08.727447 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:08.727414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:08.942655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:08.942565    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"]
Sep 09 21:47:08.945337 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:08.945308    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"
Sep 09 21:47:08.947993 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:08.947969    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lqqjx\""
Sep 09 21:47:08.956327 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:08.954653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"]
Sep 09 21:47:09.017817 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:09.017742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4262\" (UniqueName: \"kubernetes.io/projected/d148961b-65bc-4d19-ad5d-c176263bdc39-kube-api-access-j4262\") pod \"cert-manager-cainjector-559cdd8c87-x5vgp\" (UID: \"d148961b-65bc-4d19-ad5d-c176263bdc39\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"
Sep 09 21:47:09.018167 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:09.018140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d148961b-65bc-4d19-ad5d-c176263bdc39-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-x5vgp\" (UID: \"d148961b-65bc-4d19-ad5d-c176263bdc39\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"
Sep 09 21:47:09.118845 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:09.118797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4262\" (UniqueName: \"kubernetes.io/projected/d148961b-65bc-4d19-ad5d-c176263bdc39-kube-api-access-j4262\") pod \"cert-manager-cainjector-559cdd8c87-x5vgp\" (UID: \"d148961b-65bc-4d19-ad5d-c176263bdc39\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"
Sep 09 21:47:09.119008 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:09.118889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d148961b-65bc-4d19-ad5d-c176263bdc39-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-x5vgp\" (UID: \"d148961b-65bc-4d19-ad5d-c176263bdc39\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"
Sep 09 21:47:09.128150 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:09.128123    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d148961b-65bc-4d19-ad5d-c176263bdc39-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-x5vgp\" (UID: \"d148961b-65bc-4d19-ad5d-c176263bdc39\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"
Sep 09 21:47:09.128307 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:09.128179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4262\" (UniqueName: \"kubernetes.io/projected/d148961b-65bc-4d19-ad5d-c176263bdc39-kube-api-access-j4262\") pod \"cert-manager-cainjector-559cdd8c87-x5vgp\" (UID: \"d148961b-65bc-4d19-ad5d-c176263bdc39\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"
Sep 09 21:47:09.259350 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:09.259318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"
Sep 09 21:47:10.825522 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:10.825491    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:10.922929 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:10.922896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:11.075487 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:11.075321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp"]
Sep 09 21:47:11.078102 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:47:11.078075    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd148961b_65bc_4d19_ad5d_c176263bdc39.slice/crio-16609e00ab3f91c2c7cd7ee27141ea9b14e0193d8d3d83d34a4b496ee222303f WatchSource:0}: Error finding container 16609e00ab3f91c2c7cd7ee27141ea9b14e0193d8d3d83d34a4b496ee222303f: Status 404 returned error can't find the container with id 16609e00ab3f91c2c7cd7ee27141ea9b14e0193d8d3d83d34a4b496ee222303f
Sep 09 21:47:11.541668 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:11.541639    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf19da70-632f-415d-9a12-9c528cbfe09f" containerID="df2cb5c14d291e6428dcbaa42769a979c471bf85e7b84f9d89376b3700bc3297" exitCode=0
Sep 09 21:47:11.541884 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:11.541735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548" event={"ID":"cf19da70-632f-415d-9a12-9c528cbfe09f","Type":"ContainerDied","Data":"df2cb5c14d291e6428dcbaa42769a979c471bf85e7b84f9d89376b3700bc3297"}
Sep 09 21:47:11.543401 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:11.543377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz" event={"ID":"ebe58885-2cc8-4d34-9ec8-86f74e88ffe1","Type":"ContainerStarted","Data":"3b7728dc0de098abfc4b3c7ea511397f27fa8235c8334589f5747b7e5ab8837c"}
Sep 09 21:47:11.543479 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:11.543438    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz"
Sep 09 21:47:11.545066 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:11.545040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp" event={"ID":"d148961b-65bc-4d19-ad5d-c176263bdc39","Type":"ContainerStarted","Data":"073d55f9d482ff2f9337cb0c066912716adfe328ec2c6ea0b2e32c5fe6301dc3"}
Sep 09 21:47:11.545151 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:11.545072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp" event={"ID":"d148961b-65bc-4d19-ad5d-c176263bdc39","Type":"ContainerStarted","Data":"16609e00ab3f91c2c7cd7ee27141ea9b14e0193d8d3d83d34a4b496ee222303f"}
Sep 09 21:47:11.607039 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:11.606987    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x5vgp" podStartSLOduration=3.606967057 podStartE2EDuration="3.606967057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:11.586024171 +0000 UTC m=+524.482690180" watchObservedRunningTime="2026-09-09 21:47:11.606967057 +0000 UTC m=+524.503633069"
Sep 09 21:47:12.551347 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:12.551310    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf19da70-632f-415d-9a12-9c528cbfe09f" containerID="6dcfdee0a386c6a64d2e63d6780fa3b2fb3b1690171e6ba4ee90f75409518e88" exitCode=0
Sep 09 21:47:12.551737 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:12.551394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548" event={"ID":"cf19da70-632f-415d-9a12-9c528cbfe09f","Type":"ContainerDied","Data":"6dcfdee0a386c6a64d2e63d6780fa3b2fb3b1690171e6ba4ee90f75409518e88"}
Sep 09 21:47:12.569564 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:12.569508    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz" podStartSLOduration=3.205599822 podStartE2EDuration="6.569492708s" totalImagesPullingTime="3.363892886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:11.606409654 +0000 UTC m=+524.503075662" watchObservedRunningTime="2026-09-09 21:47:12.569492708 +0000 UTC m=+525.466158719"
Sep 09 21:47:13.683116 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:13.683089    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:13.760091 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:13.760061    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-bundle\") pod \"cf19da70-632f-415d-9a12-9c528cbfe09f\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") "
Sep 09 21:47:13.760285 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:13.760118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm9m5\" (UniqueName: \"kubernetes.io/projected/cf19da70-632f-415d-9a12-9c528cbfe09f-kube-api-access-dm9m5\") pod \"cf19da70-632f-415d-9a12-9c528cbfe09f\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") "
Sep 09 21:47:13.760285 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:13.760137    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-util\") pod \"cf19da70-632f-415d-9a12-9c528cbfe09f\" (UID: \"cf19da70-632f-415d-9a12-9c528cbfe09f\") "
Sep 09 21:47:13.760520 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:13.760496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-bundle" (OuterVolumeSpecName: "bundle") pod "cf19da70-632f-415d-9a12-9c528cbfe09f" (UID: "cf19da70-632f-415d-9a12-9c528cbfe09f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:13.762312 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:13.762285    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf19da70-632f-415d-9a12-9c528cbfe09f-kube-api-access-dm9m5" (OuterVolumeSpecName: "kube-api-access-dm9m5") pod "cf19da70-632f-415d-9a12-9c528cbfe09f" (UID: "cf19da70-632f-415d-9a12-9c528cbfe09f"). InnerVolumeSpecName "kube-api-access-dm9m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:13.764280 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:13.764257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-util" (OuterVolumeSpecName: "util") pod "cf19da70-632f-415d-9a12-9c528cbfe09f" (UID: "cf19da70-632f-415d-9a12-9c528cbfe09f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:13.861212 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:13.861125    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:47:13.861212 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:13.861155    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf19da70-632f-415d-9a12-9c528cbfe09f-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:47:13.861212 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:13.861165    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm9m5\" (UniqueName: \"kubernetes.io/projected/cf19da70-632f-415d-9a12-9c528cbfe09f-kube-api-access-dm9m5\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:47:14.561938 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:14.561906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548" event={"ID":"cf19da70-632f-415d-9a12-9c528cbfe09f","Type":"ContainerDied","Data":"c3e6d14e1c8f253308d2857752a24da87805095fc2a0a8003c2197f918aed3de"}
Sep 09 21:47:14.561938 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:14.561939    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsk548"
Sep 09 21:47:14.562137 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:14.561955    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3e6d14e1c8f253308d2857752a24da87805095fc2a0a8003c2197f918aed3de"
Sep 09 21:47:15.869294 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:15.869266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:16.752755 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:16.752710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:17.553739 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:17.553713    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-q8tlz"
Sep 09 21:47:18.410865 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:18.410830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:18.509361 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:18.509326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:19.541110 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.541075    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jgdzx"]
Sep 09 21:47:19.541488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.541405    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf19da70-632f-415d-9a12-9c528cbfe09f" containerName="util"
Sep 09 21:47:19.541488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.541415    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf19da70-632f-415d-9a12-9c528cbfe09f" containerName="util"
Sep 09 21:47:19.541488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.541446    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf19da70-632f-415d-9a12-9c528cbfe09f" containerName="pull"
Sep 09 21:47:19.541488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.541451    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf19da70-632f-415d-9a12-9c528cbfe09f" containerName="pull"
Sep 09 21:47:19.541488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.541462    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf19da70-632f-415d-9a12-9c528cbfe09f" containerName="extract"
Sep 09 21:47:19.541488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.541469    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf19da70-632f-415d-9a12-9c528cbfe09f" containerName="extract"
Sep 09 21:47:19.541682 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.541523    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf19da70-632f-415d-9a12-9c528cbfe09f" containerName="extract"
Sep 09 21:47:19.545302 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.545285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jgdzx"
Sep 09 21:47:19.548294 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.548273    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ngnqj\""
Sep 09 21:47:19.559105 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.559082    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jgdzx"]
Sep 09 21:47:19.611241 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.611205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjpf2\" (UniqueName: \"kubernetes.io/projected/31db26af-8afa-46d9-8556-07680f663c34-kube-api-access-pjpf2\") pod \"cert-manager-55b4d954d5-jgdzx\" (UID: \"31db26af-8afa-46d9-8556-07680f663c34\") " pod="cert-manager/cert-manager-55b4d954d5-jgdzx"
Sep 09 21:47:19.611413 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.611250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31db26af-8afa-46d9-8556-07680f663c34-bound-sa-token\") pod \"cert-manager-55b4d954d5-jgdzx\" (UID: \"31db26af-8afa-46d9-8556-07680f663c34\") " pod="cert-manager/cert-manager-55b4d954d5-jgdzx"
Sep 09 21:47:19.712255 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.712215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjpf2\" (UniqueName: \"kubernetes.io/projected/31db26af-8afa-46d9-8556-07680f663c34-kube-api-access-pjpf2\") pod \"cert-manager-55b4d954d5-jgdzx\" (UID: \"31db26af-8afa-46d9-8556-07680f663c34\") " pod="cert-manager/cert-manager-55b4d954d5-jgdzx"
Sep 09 21:47:19.712392 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.712271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31db26af-8afa-46d9-8556-07680f663c34-bound-sa-token\") pod \"cert-manager-55b4d954d5-jgdzx\" (UID: \"31db26af-8afa-46d9-8556-07680f663c34\") " pod="cert-manager/cert-manager-55b4d954d5-jgdzx"
Sep 09 21:47:19.721215 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.721187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31db26af-8afa-46d9-8556-07680f663c34-bound-sa-token\") pod \"cert-manager-55b4d954d5-jgdzx\" (UID: \"31db26af-8afa-46d9-8556-07680f663c34\") " pod="cert-manager/cert-manager-55b4d954d5-jgdzx"
Sep 09 21:47:19.721515 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.721495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjpf2\" (UniqueName: \"kubernetes.io/projected/31db26af-8afa-46d9-8556-07680f663c34-kube-api-access-pjpf2\") pod \"cert-manager-55b4d954d5-jgdzx\" (UID: \"31db26af-8afa-46d9-8556-07680f663c34\") " pod="cert-manager/cert-manager-55b4d954d5-jgdzx"
Sep 09 21:47:19.853546 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:19.853462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jgdzx"
Sep 09 21:47:20.129348 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:20.129325    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jgdzx"]
Sep 09 21:47:20.131799 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:47:20.131751    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31db26af_8afa_46d9_8556_07680f663c34.slice/crio-edfa3ba1a5ec2e619e8bdf1ad6b801a1abae6d9ec23795a86c864adab5447162 WatchSource:0}: Error finding container edfa3ba1a5ec2e619e8bdf1ad6b801a1abae6d9ec23795a86c864adab5447162: Status 404 returned error can't find the container with id edfa3ba1a5ec2e619e8bdf1ad6b801a1abae6d9ec23795a86c864adab5447162
Sep 09 21:47:20.589999 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:20.589963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jgdzx" event={"ID":"31db26af-8afa-46d9-8556-07680f663c34","Type":"ContainerStarted","Data":"82f66a281646ad69b3d455bfc7bdcf1fa8d6202500fde900f3e4cda17a63091d"}
Sep 09 21:47:20.589999 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:20.590004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jgdzx" event={"ID":"31db26af-8afa-46d9-8556-07680f663c34","Type":"ContainerStarted","Data":"edfa3ba1a5ec2e619e8bdf1ad6b801a1abae6d9ec23795a86c864adab5447162"}
Sep 09 21:47:20.611529 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:20.611476    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jgdzx" podStartSLOduration=1.61146059 podStartE2EDuration="1.61146059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:20.611065249 +0000 UTC m=+533.507731257" watchObservedRunningTime="2026-09-09 21:47:20.61146059 +0000 UTC m=+533.508126597"
Sep 09 21:47:23.509354 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:23.509322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:23.578282 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:23.578243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:25.457531 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.457497    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"]
Sep 09 21:47:25.511000 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.510959    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"]
Sep 09 21:47:25.511000 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.510996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:25.513760 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.513734    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 21:47:25.514731 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.514708    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 21:47:25.514878 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.514756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nnpqj\""
Sep 09 21:47:25.665378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.665347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6fgk\" (UniqueName: \"kubernetes.io/projected/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-kube-api-access-q6fgk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:25.665574 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.665404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:25.665574 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.665491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:25.766664 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.766627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:25.766877 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.766821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:25.766958 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.766896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6fgk\" (UniqueName: \"kubernetes.io/projected/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-kube-api-access-q6fgk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:25.767083 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.767062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:25.767169 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.767149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:25.776782 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.776742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6fgk\" (UniqueName: \"kubernetes.io/projected/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-kube-api-access-q6fgk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:25.821312 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:25.821280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:26.048445 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:26.048373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:26.119098 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:26.119073    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"]
Sep 09 21:47:26.120257 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:47:26.120215    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bd6f8c_b62a_4b4a_a64f_2ea4256d7097.slice/crio-d825d89d87f7b23e5e4fd0ef4f96c8321eb821f08877898686d5a87a06513508 WatchSource:0}: Error finding container d825d89d87f7b23e5e4fd0ef4f96c8321eb821f08877898686d5a87a06513508: Status 404 returned error can't find the container with id d825d89d87f7b23e5e4fd0ef4f96c8321eb821f08877898686d5a87a06513508
Sep 09 21:47:26.620298 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:26.620265    2563 generic.go:356] "Generic (PLEG): container finished" podID="68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" containerID="2856b5cb1d7e6427077e8799793ac869bf14b199b720ef575b0880e48e745d43" exitCode=0
Sep 09 21:47:26.620690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:26.620345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467" event={"ID":"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097","Type":"ContainerDied","Data":"2856b5cb1d7e6427077e8799793ac869bf14b199b720ef575b0880e48e745d43"}
Sep 09 21:47:26.620690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:26.620398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467" event={"ID":"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097","Type":"ContainerStarted","Data":"d825d89d87f7b23e5e4fd0ef4f96c8321eb821f08877898686d5a87a06513508"}
Sep 09 21:47:28.632581 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:28.632549    2563 generic.go:356] "Generic (PLEG): container finished" podID="68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" containerID="2a3001dff6a6bc6949b238162de73927382754790f366e4cea832f8774b67360" exitCode=0
Sep 09 21:47:28.633040 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:28.632638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467" event={"ID":"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097","Type":"ContainerDied","Data":"2a3001dff6a6bc6949b238162de73927382754790f366e4cea832f8774b67360"}
Sep 09 21:47:28.703353 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:28.703320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:28.810352 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:28.810324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:29.639429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:29.639391    2563 generic.go:356] "Generic (PLEG): container finished" podID="68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" containerID="8f7655b6db7723b8304ae7a609f734c6cc582e51e9fdc9ac2f3a16eb3949ee02" exitCode=0
Sep 09 21:47:29.639812 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:29.639444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467" event={"ID":"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097","Type":"ContainerDied","Data":"8f7655b6db7723b8304ae7a609f734c6cc582e51e9fdc9ac2f3a16eb3949ee02"}
Sep 09 21:47:30.763142 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:30.763119    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:30.914318 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:30.914219    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-bundle\") pod \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") "
Sep 09 21:47:30.914318 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:30.914292    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-util\") pod \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") "
Sep 09 21:47:30.914541 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:30.914354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6fgk\" (UniqueName: \"kubernetes.io/projected/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-kube-api-access-q6fgk\") pod \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\" (UID: \"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097\") "
Sep 09 21:47:30.915163 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:30.915127    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-bundle" (OuterVolumeSpecName: "bundle") pod "68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" (UID: "68bd6f8c-b62a-4b4a-a64f-2ea4256d7097"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:30.916487 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:30.916463    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-kube-api-access-q6fgk" (OuterVolumeSpecName: "kube-api-access-q6fgk") pod "68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" (UID: "68bd6f8c-b62a-4b4a-a64f-2ea4256d7097"). InnerVolumeSpecName "kube-api-access-q6fgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:30.920400 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:30.920368    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-util" (OuterVolumeSpecName: "util") pod "68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" (UID: "68bd6f8c-b62a-4b4a-a64f-2ea4256d7097"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:31.015146 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:31.015109    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:47:31.015146 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:31.015140    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6fgk\" (UniqueName: \"kubernetes.io/projected/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-kube-api-access-q6fgk\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:47:31.015146 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:31.015150    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68bd6f8c-b62a-4b4a-a64f-2ea4256d7097-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:47:31.650554 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:31.650518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467" event={"ID":"68bd6f8c-b62a-4b4a-a64f-2ea4256d7097","Type":"ContainerDied","Data":"d825d89d87f7b23e5e4fd0ef4f96c8321eb821f08877898686d5a87a06513508"}
Sep 09 21:47:31.650554 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:31.650539    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqw467"
Sep 09 21:47:31.650821 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:31.650575    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d825d89d87f7b23e5e4fd0ef4f96c8321eb821f08877898686d5a87a06513508"
Sep 09 21:47:41.696249 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.696218    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"]
Sep 09 21:47:41.697453 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.697417    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" containerName="extract"
Sep 09 21:47:41.697453 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.697455    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" containerName="extract"
Sep 09 21:47:41.697622 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.697498    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" containerName="util"
Sep 09 21:47:41.697622 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.697506    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" containerName="util"
Sep 09 21:47:41.697622 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.697525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" containerName="pull"
Sep 09 21:47:41.697622 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.697533    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" containerName="pull"
Sep 09 21:47:41.697826 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.697698    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bd6f8c-b62a-4b4a-a64f-2ea4256d7097" containerName="extract"
Sep 09 21:47:41.702551 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.702023    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:41.705083 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.705064    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 21:47:41.706153 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.706123    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nnpqj\""
Sep 09 21:47:41.706262 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.706158    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 21:47:41.709747 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.709725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"]
Sep 09 21:47:41.801729 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.801694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:41.801913 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.801745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq5bp\" (UniqueName: \"kubernetes.io/projected/30472d78-30ee-456c-97a4-f4b432950b25-kube-api-access-kq5bp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:41.801913 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.801832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:41.853192 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.853160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:41.902284 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.902253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:41.902435 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.902307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq5bp\" (UniqueName: \"kubernetes.io/projected/30472d78-30ee-456c-97a4-f4b432950b25-kube-api-access-kq5bp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:41.902435 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.902363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:41.902617 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.902599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:41.902690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.902671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:41.917653 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:41.917621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq5bp\" (UniqueName: \"kubernetes.io/projected/30472d78-30ee-456c-97a4-f4b432950b25-kube-api-access-kq5bp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:42.011977 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.011946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:42.283146 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.283078    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"]
Sep 09 21:47:42.285846 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:47:42.285817    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30472d78_30ee_456c_97a4_f4b432950b25.slice/crio-f670b5a41850fc3877dd992726b133ff863814c159d77bd98ab224add21e062d WatchSource:0}: Error finding container f670b5a41850fc3877dd992726b133ff863814c159d77bd98ab224add21e062d: Status 404 returned error can't find the container with id f670b5a41850fc3877dd992726b133ff863814c159d77bd98ab224add21e062d
Sep 09 21:47:42.702440 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.702347    2563 generic.go:356] "Generic (PLEG): container finished" podID="30472d78-30ee-456c-97a4-f4b432950b25" containerID="cdc451eeb69c087909998032b9c2b7f50de628c36f648cae5fd3d5dd10a08923" exitCode=0
Sep 09 21:47:42.702858 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.702436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t" event={"ID":"30472d78-30ee-456c-97a4-f4b432950b25","Type":"ContainerDied","Data":"cdc451eeb69c087909998032b9c2b7f50de628c36f648cae5fd3d5dd10a08923"}
Sep 09 21:47:42.702858 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.702490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t" event={"ID":"30472d78-30ee-456c-97a4-f4b432950b25","Type":"ContainerStarted","Data":"f670b5a41850fc3877dd992726b133ff863814c159d77bd98ab224add21e062d"}
Sep 09 21:47:42.853185 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.853149    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"]
Sep 09 21:47:42.855729 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.855706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:42.858666 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.858645    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7txkq\""
Sep 09 21:47:42.858846 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.858831    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 21:47:42.858901 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.858865    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 21:47:42.859127 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.859111    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 21:47:42.859379 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.859366    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 21:47:42.871876 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.871850    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"]
Sep 09 21:47:42.909405 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.909371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99905997-d07a-4e23-9a27-67c395cf1347-webhook-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:42.909562 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.909426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99905997-d07a-4e23-9a27-67c395cf1347-manifests\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:42.909562 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.909445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99905997-d07a-4e23-9a27-67c395cf1347-tmp\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:42.909562 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.909493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99905997-d07a-4e23-9a27-67c395cf1347-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:42.909562 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:42.909510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f45nd\" (UniqueName: \"kubernetes.io/projected/99905997-d07a-4e23-9a27-67c395cf1347-kube-api-access-f45nd\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.010651 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.010610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99905997-d07a-4e23-9a27-67c395cf1347-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.010879 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.010663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f45nd\" (UniqueName: \"kubernetes.io/projected/99905997-d07a-4e23-9a27-67c395cf1347-kube-api-access-f45nd\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.010879 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.010715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99905997-d07a-4e23-9a27-67c395cf1347-webhook-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.010879 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.010759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99905997-d07a-4e23-9a27-67c395cf1347-manifests\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.010879 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.010820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99905997-d07a-4e23-9a27-67c395cf1347-tmp\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.011200 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.011173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99905997-d07a-4e23-9a27-67c395cf1347-manifests\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.013135 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.013104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99905997-d07a-4e23-9a27-67c395cf1347-tmp\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.013328 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.013310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99905997-d07a-4e23-9a27-67c395cf1347-webhook-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.013372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.013325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99905997-d07a-4e23-9a27-67c395cf1347-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.022635 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.022608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f45nd\" (UniqueName: \"kubernetes.io/projected/99905997-d07a-4e23-9a27-67c395cf1347-kube-api-access-f45nd\") pod \"opendatahub-operator-controller-manager-56bc947947-stwkq\" (UID: \"99905997-d07a-4e23-9a27-67c395cf1347\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.060484 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.060453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:43.166258 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.166217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:43.176977 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.176948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:43.274054 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.273975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:43.373955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.373928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:43.445826 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.445680    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"]
Sep 09 21:47:43.448585 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:47:43.448558    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99905997_d07a_4e23_9a27_67c395cf1347.slice/crio-b7633e0956c4109d8b7f45b154c72f28a43154d56994eee1a7a2216486a5d8ae WatchSource:0}: Error finding container b7633e0956c4109d8b7f45b154c72f28a43154d56994eee1a7a2216486a5d8ae: Status 404 returned error can't find the container with id b7633e0956c4109d8b7f45b154c72f28a43154d56994eee1a7a2216486a5d8ae
Sep 09 21:47:43.709564 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.709470    2563 generic.go:356] "Generic (PLEG): container finished" podID="30472d78-30ee-456c-97a4-f4b432950b25" containerID="0d193e1c946c14b8942f93a507656a16ace1766ac8e52526a9fe23f588f88ebc" exitCode=0
Sep 09 21:47:43.710041 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.709572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t" event={"ID":"30472d78-30ee-456c-97a4-f4b432950b25","Type":"ContainerDied","Data":"0d193e1c946c14b8942f93a507656a16ace1766ac8e52526a9fe23f588f88ebc"}
Sep 09 21:47:43.711187 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:43.711166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq" event={"ID":"99905997-d07a-4e23-9a27-67c395cf1347","Type":"ContainerStarted","Data":"b7633e0956c4109d8b7f45b154c72f28a43154d56994eee1a7a2216486a5d8ae"}
Sep 09 21:47:44.720680 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:44.720607    2563 generic.go:356] "Generic (PLEG): container finished" podID="30472d78-30ee-456c-97a4-f4b432950b25" containerID="44b9d1ce9c5072148d053f68a3d690bed2ed5052b5370f585e1d4a1ee3c05439" exitCode=0
Sep 09 21:47:44.721117 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:44.720695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t" event={"ID":"30472d78-30ee-456c-97a4-f4b432950b25","Type":"ContainerDied","Data":"44b9d1ce9c5072148d053f68a3d690bed2ed5052b5370f585e1d4a1ee3c05439"}
Sep 09 21:47:46.412070 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.412044    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:46.437375 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.437352    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq5bp\" (UniqueName: \"kubernetes.io/projected/30472d78-30ee-456c-97a4-f4b432950b25-kube-api-access-kq5bp\") pod \"30472d78-30ee-456c-97a4-f4b432950b25\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") "
Sep 09 21:47:46.437469 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.437391    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-bundle\") pod \"30472d78-30ee-456c-97a4-f4b432950b25\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") "
Sep 09 21:47:46.437469 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.437444    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-util\") pod \"30472d78-30ee-456c-97a4-f4b432950b25\" (UID: \"30472d78-30ee-456c-97a4-f4b432950b25\") "
Sep 09 21:47:46.438194 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.438168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-bundle" (OuterVolumeSpecName: "bundle") pod "30472d78-30ee-456c-97a4-f4b432950b25" (UID: "30472d78-30ee-456c-97a4-f4b432950b25"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:46.440234 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.440206    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30472d78-30ee-456c-97a4-f4b432950b25-kube-api-access-kq5bp" (OuterVolumeSpecName: "kube-api-access-kq5bp") pod "30472d78-30ee-456c-97a4-f4b432950b25" (UID: "30472d78-30ee-456c-97a4-f4b432950b25"). InnerVolumeSpecName "kube-api-access-kq5bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:46.443432 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.443388    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-util" (OuterVolumeSpecName: "util") pod "30472d78-30ee-456c-97a4-f4b432950b25" (UID: "30472d78-30ee-456c-97a4-f4b432950b25"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:46.538356 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.538326    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:47:46.538356 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.538353    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq5bp\" (UniqueName: \"kubernetes.io/projected/30472d78-30ee-456c-97a4-f4b432950b25-kube-api-access-kq5bp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:47:46.538356 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.538362    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30472d78-30ee-456c-97a4-f4b432950b25-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:47:46.733345 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.733270    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t"
Sep 09 21:47:46.733503 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.733269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04tj4t" event={"ID":"30472d78-30ee-456c-97a4-f4b432950b25","Type":"ContainerDied","Data":"f670b5a41850fc3877dd992726b133ff863814c159d77bd98ab224add21e062d"}
Sep 09 21:47:46.733503 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.733404    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f670b5a41850fc3877dd992726b133ff863814c159d77bd98ab224add21e062d"
Sep 09 21:47:46.735113 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:46.735082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq" event={"ID":"99905997-d07a-4e23-9a27-67c395cf1347","Type":"ContainerStarted","Data":"ff7286bd27ca5840d619a1f09b120b6db394eef082401cb0a03f1ddd99529cdf"}
Sep 09 21:47:47.742304 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:47.742260    2563 generic.go:356] "Generic (PLEG): container finished" podID="99905997-d07a-4e23-9a27-67c395cf1347" containerID="ff7286bd27ca5840d619a1f09b120b6db394eef082401cb0a03f1ddd99529cdf" exitCode=0
Sep 09 21:47:47.742692 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:47.742333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq" event={"ID":"99905997-d07a-4e23-9a27-67c395cf1347","Type":"ContainerDied","Data":"ff7286bd27ca5840d619a1f09b120b6db394eef082401cb0a03f1ddd99529cdf"}
Sep 09 21:47:48.749187 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:48.749152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq" event={"ID":"99905997-d07a-4e23-9a27-67c395cf1347","Type":"ContainerStarted","Data":"23b1c99d2b878fbf4b5c05ee2a6a9bfa5dbb8a65d244bdd87a9d6f8276bd0f03"}
Sep 09 21:47:48.749579 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:48.749243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:47:48.797794 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:48.797713    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq" podStartSLOduration=2.35767177 podStartE2EDuration="6.797694167s" totalImagesPullingTime="3.3555934s" totalInitContainerRuntime="1.084428997s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:48.795982202 +0000 UTC m=+561.692648211" watchObservedRunningTime="2026-09-09 21:47:48.797694167 +0000 UTC m=+561.694360176"
Sep 09 21:47:52.406867 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:52.406836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:52.564229 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:52.564197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:54.629152 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:54.629116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:54.781918 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:54.781888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:59.757153 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:47:59.757122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-stwkq"
Sep 09 21:48:02.330759 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.330724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:02.480370 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.480332    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:02.801447 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.801414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:02.875574 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.875540    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"]
Sep 09 21:48:02.875912 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.875897    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30472d78-30ee-456c-97a4-f4b432950b25" containerName="extract"
Sep 09 21:48:02.875912 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.875913    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30472d78-30ee-456c-97a4-f4b432950b25" containerName="extract"
Sep 09 21:48:02.875998 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.875923    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30472d78-30ee-456c-97a4-f4b432950b25" containerName="pull"
Sep 09 21:48:02.875998 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.875929    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30472d78-30ee-456c-97a4-f4b432950b25" containerName="pull"
Sep 09 21:48:02.875998 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.875940    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30472d78-30ee-456c-97a4-f4b432950b25" containerName="util"
Sep 09 21:48:02.875998 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.875946    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30472d78-30ee-456c-97a4-f4b432950b25" containerName="util"
Sep 09 21:48:02.876130 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.876002    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="30472d78-30ee-456c-97a4-f4b432950b25" containerName="extract"
Sep 09 21:48:02.884366 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.884343    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:02.889130 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.889106    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 21:48:02.889261 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.889185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 21:48:02.890174 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.890153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9kjld\""
Sep 09 21:48:02.890174 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.890164    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 21:48:02.893990 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.893963    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 21:48:02.957550 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.957519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"]
Sep 09 21:48:02.963431 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.963398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/626afafe-e25d-4f41-b58f-86cf0cf0df6e-tmp\") pod \"kube-auth-proxy-7b49d8b8f6-m54ns\" (UID: \"626afafe-e25d-4f41-b58f-86cf0cf0df6e\") " pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:02.963599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.963453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4c5\" (UniqueName: \"kubernetes.io/projected/626afafe-e25d-4f41-b58f-86cf0cf0df6e-kube-api-access-sj4c5\") pod \"kube-auth-proxy-7b49d8b8f6-m54ns\" (UID: \"626afafe-e25d-4f41-b58f-86cf0cf0df6e\") " pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:02.963599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:02.963497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626afafe-e25d-4f41-b58f-86cf0cf0df6e-tls-certs\") pod \"kube-auth-proxy-7b49d8b8f6-m54ns\" (UID: \"626afafe-e25d-4f41-b58f-86cf0cf0df6e\") " pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:03.064708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:03.064613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/626afafe-e25d-4f41-b58f-86cf0cf0df6e-tmp\") pod \"kube-auth-proxy-7b49d8b8f6-m54ns\" (UID: \"626afafe-e25d-4f41-b58f-86cf0cf0df6e\") " pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:03.064708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:03.064658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4c5\" (UniqueName: \"kubernetes.io/projected/626afafe-e25d-4f41-b58f-86cf0cf0df6e-kube-api-access-sj4c5\") pod \"kube-auth-proxy-7b49d8b8f6-m54ns\" (UID: \"626afafe-e25d-4f41-b58f-86cf0cf0df6e\") " pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:03.064708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:03.064682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626afafe-e25d-4f41-b58f-86cf0cf0df6e-tls-certs\") pod \"kube-auth-proxy-7b49d8b8f6-m54ns\" (UID: \"626afafe-e25d-4f41-b58f-86cf0cf0df6e\") " pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:03.065026 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:48:03.064796    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 09 21:48:03.065026 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:48:03.064867    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626afafe-e25d-4f41-b58f-86cf0cf0df6e-tls-certs podName:626afafe-e25d-4f41-b58f-86cf0cf0df6e nodeName:}" failed. No retries permitted until 2026-09-09 21:48:03.564846487 +0000 UTC m=+576.461512473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/626afafe-e25d-4f41-b58f-86cf0cf0df6e-tls-certs") pod "kube-auth-proxy-7b49d8b8f6-m54ns" (UID: "626afafe-e25d-4f41-b58f-86cf0cf0df6e") : secret "kube-auth-proxy-tls" not found
Sep 09 21:48:03.067150 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:03.067128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/626afafe-e25d-4f41-b58f-86cf0cf0df6e-tmp\") pod \"kube-auth-proxy-7b49d8b8f6-m54ns\" (UID: \"626afafe-e25d-4f41-b58f-86cf0cf0df6e\") " pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:03.076933 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:03.076902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4c5\" (UniqueName: \"kubernetes.io/projected/626afafe-e25d-4f41-b58f-86cf0cf0df6e-kube-api-access-sj4c5\") pod \"kube-auth-proxy-7b49d8b8f6-m54ns\" (UID: \"626afafe-e25d-4f41-b58f-86cf0cf0df6e\") " pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:03.250818 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:03.250760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:03.569575 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:03.569534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626afafe-e25d-4f41-b58f-86cf0cf0df6e-tls-certs\") pod \"kube-auth-proxy-7b49d8b8f6-m54ns\" (UID: \"626afafe-e25d-4f41-b58f-86cf0cf0df6e\") " pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:03.572033 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:03.572009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626afafe-e25d-4f41-b58f-86cf0cf0df6e-tls-certs\") pod \"kube-auth-proxy-7b49d8b8f6-m54ns\" (UID: \"626afafe-e25d-4f41-b58f-86cf0cf0df6e\") " pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:03.797502 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:03.797456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"
Sep 09 21:48:04.064572 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.064531    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d797bc5b8-r8dpf"]
Sep 09 21:48:04.066985 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.066961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:04.067164 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.067144    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.099793 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.099745    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d797bc5b8-r8dpf"]
Sep 09 21:48:04.102165 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:48:04.102133    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626afafe_e25d_4f41_b58f_86cf0cf0df6e.slice/crio-34e7f7e49e5fd78657e8ce08c214749c7902389fc76c735029de364d103de292 WatchSource:0}: Error finding container 34e7f7e49e5fd78657e8ce08c214749c7902389fc76c735029de364d103de292: Status 404 returned error can't find the container with id 34e7f7e49e5fd78657e8ce08c214749c7902389fc76c735029de364d103de292
Sep 09 21:48:04.102408 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.102381    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns"]
Sep 09 21:48:04.174842 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.174797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-trusted-ca-bundle\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.175019 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.174868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-console-config\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.175019 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.174893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-console-serving-cert\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.175019 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.174910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-service-ca\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.175019 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.174928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-oauth-serving-cert\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.175168 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.175010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwmg\" (UniqueName: \"kubernetes.io/projected/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-kube-api-access-2vwmg\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.175168 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.175079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-console-oauth-config\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.194242 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.194207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:04.275853 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.275810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-console-oauth-config\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.276033 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.275870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-trusted-ca-bundle\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.276033 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.275911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-console-config\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.276033 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.275949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-console-serving-cert\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.276033 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.275971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-service-ca\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.276033 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.275989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-oauth-serving-cert\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.276033 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.276025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwmg\" (UniqueName: \"kubernetes.io/projected/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-kube-api-access-2vwmg\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.276894 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.276863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-trusted-ca-bundle\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.277062 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.277038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-service-ca\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.277185 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.277065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-console-config\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.277291 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.277269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-oauth-serving-cert\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.278904 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.278881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-console-serving-cert\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.278991 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.278901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-console-oauth-config\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.285677 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.285655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwmg\" (UniqueName: \"kubernetes.io/projected/79964bc9-ba89-4adf-bf5e-6ee2ac038b15-kube-api-access-2vwmg\") pod \"console-5d797bc5b8-r8dpf\" (UID: \"79964bc9-ba89-4adf-bf5e-6ee2ac038b15\") " pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.377199 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.377116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:04.668269 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.668246    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d797bc5b8-r8dpf"]
Sep 09 21:48:04.670799 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:48:04.670748    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79964bc9_ba89_4adf_bf5e_6ee2ac038b15.slice/crio-e8c63af47f41264320c0c07dd3bf0f674b11ef440521a8695328a056e8b4e970 WatchSource:0}: Error finding container e8c63af47f41264320c0c07dd3bf0f674b11ef440521a8695328a056e8b4e970: Status 404 returned error can't find the container with id e8c63af47f41264320c0c07dd3bf0f674b11ef440521a8695328a056e8b4e970
Sep 09 21:48:04.681403 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.681375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:04.778590 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.778555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:04.832339 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.832300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d797bc5b8-r8dpf" event={"ID":"79964bc9-ba89-4adf-bf5e-6ee2ac038b15","Type":"ContainerStarted","Data":"fbea2020f5478e27bfb133ac79bbf4204b3180568c174b356de4ac42fbff96f6"}
Sep 09 21:48:04.832514 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.832400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d797bc5b8-r8dpf" event={"ID":"79964bc9-ba89-4adf-bf5e-6ee2ac038b15","Type":"ContainerStarted","Data":"e8c63af47f41264320c0c07dd3bf0f674b11ef440521a8695328a056e8b4e970"}
Sep 09 21:48:04.834481 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.834451    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns" event={"ID":"626afafe-e25d-4f41-b58f-86cf0cf0df6e","Type":"ContainerStarted","Data":"34e7f7e49e5fd78657e8ce08c214749c7902389fc76c735029de364d103de292"}
Sep 09 21:48:04.855720 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:04.855662    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d797bc5b8-r8dpf" podStartSLOduration=0.855637966 podStartE2EDuration="855.637966ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:04.854245195 +0000 UTC m=+577.750911203" watchObservedRunningTime="2026-09-09 21:48:04.855637966 +0000 UTC m=+577.752303977"
Sep 09 21:48:05.952662 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:05.952625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:07.853125 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:07.853089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns" event={"ID":"626afafe-e25d-4f41-b58f-86cf0cf0df6e","Type":"ContainerStarted","Data":"cd58a5a6edfca86ae386a6a54782b14398884a545705f11ffb5ef6bc7a5afa36"}
Sep 09 21:48:07.870155 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:07.870111    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b49d8b8f6-m54ns" podStartSLOduration=2.919872649 podStartE2EDuration="5.87009684s" totalImagesPullingTime="2.950224191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:07.868031985 +0000 UTC m=+580.764697993" watchObservedRunningTime="2026-09-09 21:48:07.87009684 +0000 UTC m=+580.766762847"
Sep 09 21:48:11.565163 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:11.565134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:11.662457 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:11.662423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:11.868792 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:11.868691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:11.993231 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:11.993195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:12.278413 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:12.278381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:13.202955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.202922    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l5s9q"]
Sep 09 21:48:13.207462 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.207439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q"
Sep 09 21:48:13.210593 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.210566    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 21:48:13.210822 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.210566    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8x8lq\""
Sep 09 21:48:13.218406 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.218382    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l5s9q"]
Sep 09 21:48:13.257525 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.257488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:13.257720 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.257645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvbnv\" (UniqueName: \"kubernetes.io/projected/6a6767bd-31c2-48f0-bacd-b29eeb1a47d9-kube-api-access-zvbnv\") pod \"odh-model-controller-7998c67dc4-l5s9q\" (UID: \"6a6767bd-31c2-48f0-bacd-b29eeb1a47d9\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q"
Sep 09 21:48:13.257720 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.257688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a6767bd-31c2-48f0-bacd-b29eeb1a47d9-cert\") pod \"odh-model-controller-7998c67dc4-l5s9q\" (UID: \"6a6767bd-31c2-48f0-bacd-b29eeb1a47d9\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q"
Sep 09 21:48:13.358142 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.358109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvbnv\" (UniqueName: \"kubernetes.io/projected/6a6767bd-31c2-48f0-bacd-b29eeb1a47d9-kube-api-access-zvbnv\") pod \"odh-model-controller-7998c67dc4-l5s9q\" (UID: \"6a6767bd-31c2-48f0-bacd-b29eeb1a47d9\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q"
Sep 09 21:48:13.358279 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.358156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a6767bd-31c2-48f0-bacd-b29eeb1a47d9-cert\") pod \"odh-model-controller-7998c67dc4-l5s9q\" (UID: \"6a6767bd-31c2-48f0-bacd-b29eeb1a47d9\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q"
Sep 09 21:48:13.360668 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.360646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a6767bd-31c2-48f0-bacd-b29eeb1a47d9-cert\") pod \"odh-model-controller-7998c67dc4-l5s9q\" (UID: \"6a6767bd-31c2-48f0-bacd-b29eeb1a47d9\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q"
Sep 09 21:48:13.369465 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.369439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvbnv\" (UniqueName: \"kubernetes.io/projected/6a6767bd-31c2-48f0-bacd-b29eeb1a47d9-kube-api-access-zvbnv\") pod \"odh-model-controller-7998c67dc4-l5s9q\" (UID: \"6a6767bd-31c2-48f0-bacd-b29eeb1a47d9\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q"
Sep 09 21:48:13.376320 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.376296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:13.506887 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.506853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:13.518937 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.518903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q"
Sep 09 21:48:13.792486 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.792455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l5s9q"]
Sep 09 21:48:13.794634 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:48:13.794606    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6767bd_31c2_48f0_bacd_b29eeb1a47d9.slice/crio-08d9e9873e0a21707742c8a15274aec0acf3088b667c075269b26948436fe635 WatchSource:0}: Error finding container 08d9e9873e0a21707742c8a15274aec0acf3088b667c075269b26948436fe635: Status 404 returned error can't find the container with id 08d9e9873e0a21707742c8a15274aec0acf3088b667c075269b26948436fe635
Sep 09 21:48:13.883286 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.883248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q" event={"ID":"6a6767bd-31c2-48f0-bacd-b29eeb1a47d9","Type":"ContainerStarted","Data":"08d9e9873e0a21707742c8a15274aec0acf3088b667c075269b26948436fe635"}
Sep 09 21:48:13.888960 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.888934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:13.940706 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.940670    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zmqhp"]
Sep 09 21:48:13.945870 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.945846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp"
Sep 09 21:48:13.948739 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.948717    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 09 21:48:13.949031 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.948729    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-mwt4w\""
Sep 09 21:48:13.962627 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:13.962605    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zmqhp"]
Sep 09 21:48:14.064751 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.064654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979bdb69-b60e-49a6-92f0-35cc3746740c-tls-certs\") pod \"model-serving-api-54d89bd5fc-zmqhp\" (UID: \"979bdb69-b60e-49a6-92f0-35cc3746740c\") " pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp"
Sep 09 21:48:14.064936 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.064791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn5s6\" (UniqueName: \"kubernetes.io/projected/979bdb69-b60e-49a6-92f0-35cc3746740c-kube-api-access-cn5s6\") pod \"model-serving-api-54d89bd5fc-zmqhp\" (UID: \"979bdb69-b60e-49a6-92f0-35cc3746740c\") " pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp"
Sep 09 21:48:14.077876 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.077847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:14.166142 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.166102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn5s6\" (UniqueName: \"kubernetes.io/projected/979bdb69-b60e-49a6-92f0-35cc3746740c-kube-api-access-cn5s6\") pod \"model-serving-api-54d89bd5fc-zmqhp\" (UID: \"979bdb69-b60e-49a6-92f0-35cc3746740c\") " pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp"
Sep 09 21:48:14.166292 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.166162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979bdb69-b60e-49a6-92f0-35cc3746740c-tls-certs\") pod \"model-serving-api-54d89bd5fc-zmqhp\" (UID: \"979bdb69-b60e-49a6-92f0-35cc3746740c\") " pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp"
Sep 09 21:48:14.168723 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.168694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979bdb69-b60e-49a6-92f0-35cc3746740c-tls-certs\") pod \"model-serving-api-54d89bd5fc-zmqhp\" (UID: \"979bdb69-b60e-49a6-92f0-35cc3746740c\") " pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp"
Sep 09 21:48:14.173595 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.173570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn5s6\" (UniqueName: \"kubernetes.io/projected/979bdb69-b60e-49a6-92f0-35cc3746740c-kube-api-access-cn5s6\") pod \"model-serving-api-54d89bd5fc-zmqhp\" (UID: \"979bdb69-b60e-49a6-92f0-35cc3746740c\") " pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp"
Sep 09 21:48:14.258409 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.258368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp"
Sep 09 21:48:14.377551 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.377460    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:14.377551 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.377538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:14.383203 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.383176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:14.536461 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.536437    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zmqhp"]
Sep 09 21:48:14.538874 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:48:14.538843    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979bdb69_b60e_49a6_92f0_35cc3746740c.slice/crio-35e4789cb2089ce18e6d48a04018b61ec803514c5767caba3ba313b90bb37d47 WatchSource:0}: Error finding container 35e4789cb2089ce18e6d48a04018b61ec803514c5767caba3ba313b90bb37d47: Status 404 returned error can't find the container with id 35e4789cb2089ce18e6d48a04018b61ec803514c5767caba3ba313b90bb37d47
Sep 09 21:48:14.889483 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.889445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp" event={"ID":"979bdb69-b60e-49a6-92f0-35cc3746740c","Type":"ContainerStarted","Data":"35e4789cb2089ce18e6d48a04018b61ec803514c5767caba3ba313b90bb37d47"}
Sep 09 21:48:14.893498 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.893467    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d797bc5b8-r8dpf"
Sep 09 21:48:14.939545 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:14.939509    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f4c554954-6jsl5"]
Sep 09 21:48:15.041261 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:15.041056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:15.143323 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:15.143215    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:15.242503 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:15.242475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:17.688610 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:17.688569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:17.788572 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:17.788535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:17.909187 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:17.909077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q" event={"ID":"6a6767bd-31c2-48f0-bacd-b29eeb1a47d9","Type":"ContainerStarted","Data":"3a66e3d7645aa9468b05342ad83ff631a86cf003e7636a6ad364fa37749e72e0"}
Sep 09 21:48:17.909379 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:17.909310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q"
Sep 09 21:48:17.911582 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:17.911534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp" event={"ID":"979bdb69-b60e-49a6-92f0-35cc3746740c","Type":"ContainerStarted","Data":"711de10ffd31955288769014f604b692439db53990e0b7a5b51352d38953dfc3"}
Sep 09 21:48:17.911732 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:17.911628    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp"
Sep 09 21:48:17.930469 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:17.930419    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q" podStartSLOduration=1.128708351 podStartE2EDuration="4.930402013s" totalImagesPullingTime="3.801693662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:17.928500021 +0000 UTC m=+590.825166030" watchObservedRunningTime="2026-09-09 21:48:17.930402013 +0000 UTC m=+590.827068014"
Sep 09 21:48:17.959661 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:17.959591    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp" podStartSLOduration=1.9561751699999999 podStartE2EDuration="4.959568935s" totalImagesPullingTime="3.003393765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:17.958231833 +0000 UTC m=+590.854897842" watchObservedRunningTime="2026-09-09 21:48:17.959568935 +0000 UTC m=+590.856234945"
Sep 09 21:48:19.638136 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.638100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:19.704981 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.704945    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"]
Sep 09 21:48:19.708851 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.708832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.710552 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.710529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:19.712708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.712685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-72vr8\""
Sep 09 21:48:19.712708 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.712697    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 21:48:19.712896 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.712753    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:48:19.712896 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.712685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 21:48:19.712896 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.712685    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 21:48:19.712896 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.712828    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 21:48:19.717829 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.717806    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"]
Sep 09 21:48:19.819195 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.819151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4e5b599-6060-4e81-88bd-574c218c7bb1-cert\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.819384 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.819239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29qtv\" (UniqueName: \"kubernetes.io/projected/f4e5b599-6060-4e81-88bd-574c218c7bb1-kube-api-access-29qtv\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.819384 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.819284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4e5b599-6060-4e81-88bd-574c218c7bb1-metrics-cert\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.819502 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.819375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4e5b599-6060-4e81-88bd-574c218c7bb1-manager-config\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.920170 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.920095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4e5b599-6060-4e81-88bd-574c218c7bb1-metrics-cert\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.920170 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.920143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4e5b599-6060-4e81-88bd-574c218c7bb1-manager-config\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.920395 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.920196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4e5b599-6060-4e81-88bd-574c218c7bb1-cert\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.920395 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.920242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29qtv\" (UniqueName: \"kubernetes.io/projected/f4e5b599-6060-4e81-88bd-574c218c7bb1-kube-api-access-29qtv\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.920983 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.920959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4e5b599-6060-4e81-88bd-574c218c7bb1-manager-config\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.922732 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.922707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4e5b599-6060-4e81-88bd-574c218c7bb1-metrics-cert\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.922906 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.922888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4e5b599-6060-4e81-88bd-574c218c7bb1-cert\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:19.929675 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:19.929645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29qtv\" (UniqueName: \"kubernetes.io/projected/f4e5b599-6060-4e81-88bd-574c218c7bb1-kube-api-access-29qtv\") pod \"lws-controller-manager-798b8d4d4-wq4x7\" (UID: \"f4e5b599-6060-4e81-88bd-574c218c7bb1\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:20.019721 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:20.019688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:20.296943 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:20.296903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"]
Sep 09 21:48:20.299263 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:48:20.299234    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e5b599_6060_4e81_88bd_574c218c7bb1.slice/crio-81c64a6775c961ddef074a860ecd79db452dcd5157e60bd1cd6ad84059d9a5c4 WatchSource:0}: Error finding container 81c64a6775c961ddef074a860ecd79db452dcd5157e60bd1cd6ad84059d9a5c4: Status 404 returned error can't find the container with id 81c64a6775c961ddef074a860ecd79db452dcd5157e60bd1cd6ad84059d9a5c4
Sep 09 21:48:20.928641 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:20.928605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7" event={"ID":"f4e5b599-6060-4e81-88bd-574c218c7bb1","Type":"ContainerStarted","Data":"81c64a6775c961ddef074a860ecd79db452dcd5157e60bd1cd6ad84059d9a5c4"}
Sep 09 21:48:21.935894 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:21.935858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7" event={"ID":"f4e5b599-6060-4e81-88bd-574c218c7bb1","Type":"ContainerStarted","Data":"3d688283a01c487d4e0800daaaa88b42a98b9afb757a2748961c1f0f219aa0a5"}
Sep 09 21:48:21.936288 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:21.936003    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:21.960262 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:21.960215    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7" podStartSLOduration=1.473447983 podStartE2EDuration="2.960201039s" totalImagesPullingTime="1.486753056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:21.959185078 +0000 UTC m=+594.855851087" watchObservedRunningTime="2026-09-09 21:48:21.960201039 +0000 UTC m=+594.856867046"
Sep 09 21:48:23.867108 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:23.867075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:23.967550 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:23.967513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:24.060562 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:24.060527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:24.306277 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:24.306246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:25.156546 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:25.156505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:25.282158 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:25.282130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:25.399341 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:25.399277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:27.526929 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:27.526902    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:48:27.529861 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:27.529837    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:48:28.919088 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:28.919045    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-l5s9q"
Sep 09 21:48:28.921942 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:28.921920    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zmqhp"
Sep 09 21:48:32.944252 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:32.944220    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-wq4x7"
Sep 09 21:48:39.965517 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:39.965481    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f4c554954-6jsl5" podUID="f55e5f7e-f2cd-45e7-bb97-866da2074892" containerName="console" containerID="cri-o://707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51" gracePeriod=15
Sep 09 21:48:40.229250 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.229167    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:48:40.311588 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.311554    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-config\") pod \"f55e5f7e-f2cd-45e7-bb97-866da2074892\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") "
Sep 09 21:48:40.311588 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.311592    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-service-ca\") pod \"f55e5f7e-f2cd-45e7-bb97-866da2074892\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") "
Sep 09 21:48:40.311876 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.311627    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-trusted-ca-bundle\") pod \"f55e5f7e-f2cd-45e7-bb97-866da2074892\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") "
Sep 09 21:48:40.311876 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.311665    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-oauth-serving-cert\") pod \"f55e5f7e-f2cd-45e7-bb97-866da2074892\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") "
Sep 09 21:48:40.311876 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.311697    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-serving-cert\") pod \"f55e5f7e-f2cd-45e7-bb97-866da2074892\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") "
Sep 09 21:48:40.311876 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.311722    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-oauth-config\") pod \"f55e5f7e-f2cd-45e7-bb97-866da2074892\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") "
Sep 09 21:48:40.311876 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.311764    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb9qb\" (UniqueName: \"kubernetes.io/projected/f55e5f7e-f2cd-45e7-bb97-866da2074892-kube-api-access-mb9qb\") pod \"f55e5f7e-f2cd-45e7-bb97-866da2074892\" (UID: \"f55e5f7e-f2cd-45e7-bb97-866da2074892\") "
Sep 09 21:48:40.312184 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.312042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-config" (OuterVolumeSpecName: "console-config") pod "f55e5f7e-f2cd-45e7-bb97-866da2074892" (UID: "f55e5f7e-f2cd-45e7-bb97-866da2074892"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:40.312184 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.312118    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f55e5f7e-f2cd-45e7-bb97-866da2074892" (UID: "f55e5f7e-f2cd-45e7-bb97-866da2074892"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:40.312184 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.312137    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-service-ca" (OuterVolumeSpecName: "service-ca") pod "f55e5f7e-f2cd-45e7-bb97-866da2074892" (UID: "f55e5f7e-f2cd-45e7-bb97-866da2074892"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:40.312184 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.312144    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f55e5f7e-f2cd-45e7-bb97-866da2074892" (UID: "f55e5f7e-f2cd-45e7-bb97-866da2074892"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:48:40.314197 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.314168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55e5f7e-f2cd-45e7-bb97-866da2074892-kube-api-access-mb9qb" (OuterVolumeSpecName: "kube-api-access-mb9qb") pod "f55e5f7e-f2cd-45e7-bb97-866da2074892" (UID: "f55e5f7e-f2cd-45e7-bb97-866da2074892"). InnerVolumeSpecName "kube-api-access-mb9qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:40.314283 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.314199    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f55e5f7e-f2cd-45e7-bb97-866da2074892" (UID: "f55e5f7e-f2cd-45e7-bb97-866da2074892"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:48:40.314283 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.314216    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f55e5f7e-f2cd-45e7-bb97-866da2074892" (UID: "f55e5f7e-f2cd-45e7-bb97-866da2074892"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:48:40.412720 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.412683    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-trusted-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:48:40.412720 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.412713    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-oauth-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:48:40.412720 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.412723    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-serving-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:48:40.412720 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.412731    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-oauth-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:48:40.413015 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.412740    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb9qb\" (UniqueName: \"kubernetes.io/projected/f55e5f7e-f2cd-45e7-bb97-866da2074892-kube-api-access-mb9qb\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:48:40.413015 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.412749    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-console-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:48:40.413015 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:40.412757    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f55e5f7e-f2cd-45e7-bb97-866da2074892-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:48:41.038990 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.038951    2563 generic.go:356] "Generic (PLEG): container finished" podID="f55e5f7e-f2cd-45e7-bb97-866da2074892" containerID="707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51" exitCode=0
Sep 09 21:48:41.039452 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.038999    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4c554954-6jsl5"
Sep 09 21:48:41.039452 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.039012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4c554954-6jsl5" event={"ID":"f55e5f7e-f2cd-45e7-bb97-866da2074892","Type":"ContainerDied","Data":"707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51"}
Sep 09 21:48:41.039452 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.039066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4c554954-6jsl5" event={"ID":"f55e5f7e-f2cd-45e7-bb97-866da2074892","Type":"ContainerDied","Data":"51d0fb0592d45060b2a7919c4594712ea9f6b3c764b59c727690340c0c66c09c"}
Sep 09 21:48:41.039452 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.039094    2563 scope.go:117] "RemoveContainer" containerID="707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51"
Sep 09 21:48:41.048359 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.048340    2563 scope.go:117] "RemoveContainer" containerID="707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51"
Sep 09 21:48:41.048618 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:48:41.048600    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51\": container with ID starting with 707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51 not found: ID does not exist" containerID="707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51"
Sep 09 21:48:41.048665 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.048626    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51"} err="failed to get container status \"707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51\": rpc error: code = NotFound desc = could not find container \"707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51\": container with ID starting with 707829c156f0bcad4f62e5332e6eaf7679b12c3ae7907d61168dd9bec1294d51 not found: ID does not exist"
Sep 09 21:48:41.062493 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.062458    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f4c554954-6jsl5"]
Sep 09 21:48:41.067554 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.067531    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f4c554954-6jsl5"]
Sep 09 21:48:41.340233 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.340150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:41.605896 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:48:41.605818    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f55e5f7e-f2cd-45e7-bb97-866da2074892" path="/var/lib/kubelet/pods/f55e5f7e-f2cd-45e7-bb97-866da2074892/volumes"
Sep 09 21:49:14.338793 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:14.338670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:14.432872 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:14.432840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:18.282750 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.282718    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"]
Sep 09 21:49:18.283185 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.283082    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55e5f7e-f2cd-45e7-bb97-866da2074892" containerName="console"
Sep 09 21:49:18.283185 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.283093    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55e5f7e-f2cd-45e7-bb97-866da2074892" containerName="console"
Sep 09 21:49:18.283185 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.283176    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55e5f7e-f2cd-45e7-bb97-866da2074892" containerName="console"
Sep 09 21:49:18.287694 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.287676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.290798 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.290761    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 21:49:18.290920 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.290763    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 21:49:18.291847 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.291829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nnpqj\""
Sep 09 21:49:18.296473 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.296452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"]
Sep 09 21:49:18.377707 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.377670    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"]
Sep 09 21:49:18.380410 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.380388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.390167 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.390138    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"]
Sep 09 21:49:18.453741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.453698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.453945 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.453748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.453945 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.453804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.453945 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.453836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmcfq\" (UniqueName: \"kubernetes.io/projected/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-kube-api-access-jmcfq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.453945 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.453871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jwt\" (UniqueName: \"kubernetes.io/projected/77428fbf-b127-43d5-97e6-66e76a7c7ea5-kube-api-access-q6jwt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.453945 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.453898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.478174 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.478135    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"]
Sep 09 21:49:18.480959 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.480941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.490335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.490309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"]
Sep 09 21:49:18.555054 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.554960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.555054 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.555054 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.555351 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmcfq\" (UniqueName: \"kubernetes.io/projected/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-kube-api-access-jmcfq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.555351 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwln\" (UniqueName: \"kubernetes.io/projected/c447ac3e-70a6-4da4-9bdb-df938a706654-kube-api-access-kqwln\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.555351 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jwt\" (UniqueName: \"kubernetes.io/projected/77428fbf-b127-43d5-97e6-66e76a7c7ea5-kube-api-access-q6jwt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.555351 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.555351 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.555558 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.555558 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.555558 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.555656 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.555690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.555672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.565015 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.564989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmcfq\" (UniqueName: \"kubernetes.io/projected/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-kube-api-access-jmcfq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.565135 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.565037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jwt\" (UniqueName: \"kubernetes.io/projected/77428fbf-b127-43d5-97e6-66e76a7c7ea5-kube-api-access-q6jwt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.584271 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.584241    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"]
Sep 09 21:49:18.586835 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.586821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.596663 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.596642    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"]
Sep 09 21:49:18.598649 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.598086    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:18.656674 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.656636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.656823 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.656754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.656890 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.656857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt2bz\" (UniqueName: \"kubernetes.io/projected/55f65f76-81da-43d4-823d-b0c8d96fa262-kube-api-access-qt2bz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.656980 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.656955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.657097 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.657007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwln\" (UniqueName: \"kubernetes.io/projected/c447ac3e-70a6-4da4-9bdb-df938a706654-kube-api-access-kqwln\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.657097 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.657066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.657811 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.657758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.658085 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.658065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.666398 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.666366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwln\" (UniqueName: \"kubernetes.io/projected/c447ac3e-70a6-4da4-9bdb-df938a706654-kube-api-access-kqwln\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.690957 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.690929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:18.758232 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.757611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.758232 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.757646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.758232 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.757664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt2bz\" (UniqueName: \"kubernetes.io/projected/55f65f76-81da-43d4-823d-b0c8d96fa262-kube-api-access-qt2bz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.758232 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.758180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.758232 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.758179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.767502 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.767476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt2bz\" (UniqueName: \"kubernetes.io/projected/55f65f76-81da-43d4-823d-b0c8d96fa262-kube-api-access-qt2bz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.790420 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.790387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:18.876927 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.876899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"]
Sep 09 21:49:18.878613 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:49:18.878589    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77428fbf_b127_43d5_97e6_66e76a7c7ea5.slice/crio-78bbb38c68b5bd768ddd48b73ff23b9ba756a1eb79acb5f2bfa42c3789283832 WatchSource:0}: Error finding container 78bbb38c68b5bd768ddd48b73ff23b9ba756a1eb79acb5f2bfa42c3789283832: Status 404 returned error can't find the container with id 78bbb38c68b5bd768ddd48b73ff23b9ba756a1eb79acb5f2bfa42c3789283832
Sep 09 21:49:18.897113 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.897083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:18.971131 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:18.971051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"]
Sep 09 21:49:18.972409 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:49:18.972379    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f58c891_a8f7_43ac_a83e_fd9fa08e2a30.slice/crio-57b7996f8845c50404b2047fdac423b53e3ca0983830b7b1c711e8a8ebea8971 WatchSource:0}: Error finding container 57b7996f8845c50404b2047fdac423b53e3ca0983830b7b1c711e8a8ebea8971: Status 404 returned error can't find the container with id 57b7996f8845c50404b2047fdac423b53e3ca0983830b7b1c711e8a8ebea8971
Sep 09 21:49:19.066809 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.066663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"]
Sep 09 21:49:19.073001 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:49:19.072972    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc447ac3e_70a6_4da4_9bdb_df938a706654.slice/crio-c665b6c56be056cc74fc69b44715c318fa989ff6ee4cb4017bd09a7a53e53c34 WatchSource:0}: Error finding container c665b6c56be056cc74fc69b44715c318fa989ff6ee4cb4017bd09a7a53e53c34: Status 404 returned error can't find the container with id c665b6c56be056cc74fc69b44715c318fa989ff6ee4cb4017bd09a7a53e53c34
Sep 09 21:49:19.178231 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.178206    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"]
Sep 09 21:49:19.237047 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.237014    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" containerID="84242af8887d3f4c637edbe9e35663f6ef6a8c77433f235c9f1d0f040d70fbb1" exitCode=0
Sep 09 21:49:19.237194 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.237101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h" event={"ID":"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30","Type":"ContainerDied","Data":"84242af8887d3f4c637edbe9e35663f6ef6a8c77433f235c9f1d0f040d70fbb1"}
Sep 09 21:49:19.237194 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.237164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h" event={"ID":"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30","Type":"ContainerStarted","Data":"57b7996f8845c50404b2047fdac423b53e3ca0983830b7b1c711e8a8ebea8971"}
Sep 09 21:49:19.238230 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.238211    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:49:19.238833 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.238812    2563 generic.go:356] "Generic (PLEG): container finished" podID="77428fbf-b127-43d5-97e6-66e76a7c7ea5" containerID="4bf8b2a07009bc83a5ff527e6a9f4d413723a0f4bfd54b42d2f6161a952ab07b" exitCode=0
Sep 09 21:49:19.238903 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.238888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56" event={"ID":"77428fbf-b127-43d5-97e6-66e76a7c7ea5","Type":"ContainerDied","Data":"4bf8b2a07009bc83a5ff527e6a9f4d413723a0f4bfd54b42d2f6161a952ab07b"}
Sep 09 21:49:19.238963 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.238931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56" event={"ID":"77428fbf-b127-43d5-97e6-66e76a7c7ea5","Type":"ContainerStarted","Data":"78bbb38c68b5bd768ddd48b73ff23b9ba756a1eb79acb5f2bfa42c3789283832"}
Sep 09 21:49:19.240585 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.240563    2563 generic.go:356] "Generic (PLEG): container finished" podID="c447ac3e-70a6-4da4-9bdb-df938a706654" containerID="e2d03edd37c1d431db8640fded53bb55bb624fc4557e34a98fada7dabef77a53" exitCode=0
Sep 09 21:49:19.240667 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.240639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl" event={"ID":"c447ac3e-70a6-4da4-9bdb-df938a706654","Type":"ContainerDied","Data":"e2d03edd37c1d431db8640fded53bb55bb624fc4557e34a98fada7dabef77a53"}
Sep 09 21:49:19.240717 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:19.240684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl" event={"ID":"c447ac3e-70a6-4da4-9bdb-df938a706654","Type":"ContainerStarted","Data":"c665b6c56be056cc74fc69b44715c318fa989ff6ee4cb4017bd09a7a53e53c34"}
Sep 09 21:49:19.272579 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:49:19.272548    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f65f76_81da_43d4_823d_b0c8d96fa262.slice/crio-fd2f37a4bf1a8f4598b732d24212834c877b9e94b8a5aa776cb9746fffb9de7c WatchSource:0}: Error finding container fd2f37a4bf1a8f4598b732d24212834c877b9e94b8a5aa776cb9746fffb9de7c: Status 404 returned error can't find the container with id fd2f37a4bf1a8f4598b732d24212834c877b9e94b8a5aa776cb9746fffb9de7c
Sep 09 21:49:20.249497 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:20.249453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h" event={"ID":"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30","Type":"ContainerStarted","Data":"b7eeec797cb26399b401bc6f2894b09d99c78693baf563d438420ca82027ca5c"}
Sep 09 21:49:20.251522 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:20.251493    2563 generic.go:356] "Generic (PLEG): container finished" podID="55f65f76-81da-43d4-823d-b0c8d96fa262" containerID="db3c83f97a873147aebd083226050ab27d79111664536b814201bb5b7021d3d8" exitCode=0
Sep 09 21:49:20.251664 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:20.251573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh" event={"ID":"55f65f76-81da-43d4-823d-b0c8d96fa262","Type":"ContainerDied","Data":"db3c83f97a873147aebd083226050ab27d79111664536b814201bb5b7021d3d8"}
Sep 09 21:49:20.251664 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:20.251623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh" event={"ID":"55f65f76-81da-43d4-823d-b0c8d96fa262","Type":"ContainerStarted","Data":"fd2f37a4bf1a8f4598b732d24212834c877b9e94b8a5aa776cb9746fffb9de7c"}
Sep 09 21:49:21.259613 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:21.259584    2563 generic.go:356] "Generic (PLEG): container finished" podID="77428fbf-b127-43d5-97e6-66e76a7c7ea5" containerID="4bf148d1c31e1e23a46840438e5edf7638eaa652cb22b633fb8d43335f525c6b" exitCode=0
Sep 09 21:49:21.260017 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:21.259675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56" event={"ID":"77428fbf-b127-43d5-97e6-66e76a7c7ea5","Type":"ContainerDied","Data":"4bf148d1c31e1e23a46840438e5edf7638eaa652cb22b633fb8d43335f525c6b"}
Sep 09 21:49:21.261826 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:21.261802    2563 generic.go:356] "Generic (PLEG): container finished" podID="c447ac3e-70a6-4da4-9bdb-df938a706654" containerID="3b8c026055604ee620593c836c46e827a6a3686eada7e0724a26a2cf31ae2f60" exitCode=0
Sep 09 21:49:21.261929 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:21.261886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl" event={"ID":"c447ac3e-70a6-4da4-9bdb-df938a706654","Type":"ContainerDied","Data":"3b8c026055604ee620593c836c46e827a6a3686eada7e0724a26a2cf31ae2f60"}
Sep 09 21:49:21.264041 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:21.264022    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" containerID="b7eeec797cb26399b401bc6f2894b09d99c78693baf563d438420ca82027ca5c" exitCode=0
Sep 09 21:49:21.264133 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:21.264048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h" event={"ID":"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30","Type":"ContainerDied","Data":"b7eeec797cb26399b401bc6f2894b09d99c78693baf563d438420ca82027ca5c"}
Sep 09 21:49:22.271391 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:22.271350    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" containerID="afdef2f687324df47b6195af15dabdc7b8fff93e6ea4415a35827a21054dde07" exitCode=0
Sep 09 21:49:22.271900 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:22.271434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h" event={"ID":"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30","Type":"ContainerDied","Data":"afdef2f687324df47b6195af15dabdc7b8fff93e6ea4415a35827a21054dde07"}
Sep 09 21:49:22.273525 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:22.273497    2563 generic.go:356] "Generic (PLEG): container finished" podID="77428fbf-b127-43d5-97e6-66e76a7c7ea5" containerID="2b0771c85947f690fed88a6e9084495b31c9b0c1186a4ceb0d14b7a4c2b8a9da" exitCode=0
Sep 09 21:49:22.273709 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:22.273594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56" event={"ID":"77428fbf-b127-43d5-97e6-66e76a7c7ea5","Type":"ContainerDied","Data":"2b0771c85947f690fed88a6e9084495b31c9b0c1186a4ceb0d14b7a4c2b8a9da"}
Sep 09 21:49:22.275736 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:22.275713    2563 generic.go:356] "Generic (PLEG): container finished" podID="c447ac3e-70a6-4da4-9bdb-df938a706654" containerID="6681f29dca0b38578cad430e744b5b609e58e29681659da5c6199d0fd6b0e45a" exitCode=0
Sep 09 21:49:22.275867 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:22.275805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl" event={"ID":"c447ac3e-70a6-4da4-9bdb-df938a706654","Type":"ContainerDied","Data":"6681f29dca0b38578cad430e744b5b609e58e29681659da5c6199d0fd6b0e45a"}
Sep 09 21:49:22.277544 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:22.277524    2563 generic.go:356] "Generic (PLEG): container finished" podID="55f65f76-81da-43d4-823d-b0c8d96fa262" containerID="ca34b72fc22febbbf0b20b6bf2eff492f89ba5fd57cc5de1eea9f8be4a45b39f" exitCode=0
Sep 09 21:49:22.277670 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:22.277569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh" event={"ID":"55f65f76-81da-43d4-823d-b0c8d96fa262","Type":"ContainerDied","Data":"ca34b72fc22febbbf0b20b6bf2eff492f89ba5fd57cc5de1eea9f8be4a45b39f"}
Sep 09 21:49:23.285657 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.285626    2563 generic.go:356] "Generic (PLEG): container finished" podID="55f65f76-81da-43d4-823d-b0c8d96fa262" containerID="6ce39299e53e79a8d10332c47da8763ed94c42a437316f09d9ac4abb65115073" exitCode=0
Sep 09 21:49:23.286062 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.285709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh" event={"ID":"55f65f76-81da-43d4-823d-b0c8d96fa262","Type":"ContainerDied","Data":"6ce39299e53e79a8d10332c47da8763ed94c42a437316f09d9ac4abb65115073"}
Sep 09 21:49:23.428209 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.428184    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:23.475990 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.475962    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:23.479869 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.479847    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:23.601980 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.601893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-util\") pod \"c447ac3e-70a6-4da4-9bdb-df938a706654\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") "
Sep 09 21:49:23.601980 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.601928    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-bundle\") pod \"c447ac3e-70a6-4da4-9bdb-df938a706654\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") "
Sep 09 21:49:23.601980 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.601982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-bundle\") pod \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") "
Sep 09 21:49:23.602267 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.602033    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6jwt\" (UniqueName: \"kubernetes.io/projected/77428fbf-b127-43d5-97e6-66e76a7c7ea5-kube-api-access-q6jwt\") pod \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") "
Sep 09 21:49:23.602267 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.602061    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-util\") pod \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\" (UID: \"77428fbf-b127-43d5-97e6-66e76a7c7ea5\") "
Sep 09 21:49:23.602267 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.602088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-bundle\") pod \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") "
Sep 09 21:49:23.602267 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.602115    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmcfq\" (UniqueName: \"kubernetes.io/projected/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-kube-api-access-jmcfq\") pod \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") "
Sep 09 21:49:23.602267 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.602139    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-util\") pod \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\" (UID: \"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30\") "
Sep 09 21:49:23.602267 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.602186    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqwln\" (UniqueName: \"kubernetes.io/projected/c447ac3e-70a6-4da4-9bdb-df938a706654-kube-api-access-kqwln\") pod \"c447ac3e-70a6-4da4-9bdb-df938a706654\" (UID: \"c447ac3e-70a6-4da4-9bdb-df938a706654\") "
Sep 09 21:49:23.602786 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.602518    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-bundle" (OuterVolumeSpecName: "bundle") pod "77428fbf-b127-43d5-97e6-66e76a7c7ea5" (UID: "77428fbf-b127-43d5-97e6-66e76a7c7ea5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:23.602786 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.602522    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-bundle" (OuterVolumeSpecName: "bundle") pod "c447ac3e-70a6-4da4-9bdb-df938a706654" (UID: "c447ac3e-70a6-4da4-9bdb-df938a706654"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:23.602959 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.602921    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-bundle" (OuterVolumeSpecName: "bundle") pod "5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" (UID: "5f58c891-a8f7-43ac-a83e-fd9fa08e2a30"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:23.605264 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.605236    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-kube-api-access-jmcfq" (OuterVolumeSpecName: "kube-api-access-jmcfq") pod "5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" (UID: "5f58c891-a8f7-43ac-a83e-fd9fa08e2a30"). InnerVolumeSpecName "kube-api-access-jmcfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:49:23.605363 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.605344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77428fbf-b127-43d5-97e6-66e76a7c7ea5-kube-api-access-q6jwt" (OuterVolumeSpecName: "kube-api-access-q6jwt") pod "77428fbf-b127-43d5-97e6-66e76a7c7ea5" (UID: "77428fbf-b127-43d5-97e6-66e76a7c7ea5"). InnerVolumeSpecName "kube-api-access-q6jwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:49:23.605434 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.605366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c447ac3e-70a6-4da4-9bdb-df938a706654-kube-api-access-kqwln" (OuterVolumeSpecName: "kube-api-access-kqwln") pod "c447ac3e-70a6-4da4-9bdb-df938a706654" (UID: "c447ac3e-70a6-4da4-9bdb-df938a706654"). InnerVolumeSpecName "kube-api-access-kqwln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:49:23.610875 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.610833    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-util" (OuterVolumeSpecName: "util") pod "77428fbf-b127-43d5-97e6-66e76a7c7ea5" (UID: "77428fbf-b127-43d5-97e6-66e76a7c7ea5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:23.610956 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.610889    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-util" (OuterVolumeSpecName: "util") pod "c447ac3e-70a6-4da4-9bdb-df938a706654" (UID: "c447ac3e-70a6-4da4-9bdb-df938a706654"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:23.611574 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.611556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-util" (OuterVolumeSpecName: "util") pod "5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" (UID: "5f58c891-a8f7-43ac-a83e-fd9fa08e2a30"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:23.703870 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.703840    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jwt\" (UniqueName: \"kubernetes.io/projected/77428fbf-b127-43d5-97e6-66e76a7c7ea5-kube-api-access-q6jwt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:23.703870 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.703865    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:23.703870 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.703877    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:23.704096 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.703886    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmcfq\" (UniqueName: \"kubernetes.io/projected/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-kube-api-access-jmcfq\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:23.704096 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.703895    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f58c891-a8f7-43ac-a83e-fd9fa08e2a30-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:23.704096 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.703904    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqwln\" (UniqueName: \"kubernetes.io/projected/c447ac3e-70a6-4da4-9bdb-df938a706654-kube-api-access-kqwln\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:23.704096 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.703913    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:23.704096 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.703922    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c447ac3e-70a6-4da4-9bdb-df938a706654-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:23.704096 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:23.703929    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77428fbf-b127-43d5-97e6-66e76a7c7ea5-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:24.292818 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.292789    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56"
Sep 09 21:49:24.292818 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.292801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mnd56" event={"ID":"77428fbf-b127-43d5-97e6-66e76a7c7ea5","Type":"ContainerDied","Data":"78bbb38c68b5bd768ddd48b73ff23b9ba756a1eb79acb5f2bfa42c3789283832"}
Sep 09 21:49:24.293340 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.292883    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78bbb38c68b5bd768ddd48b73ff23b9ba756a1eb79acb5f2bfa42c3789283832"
Sep 09 21:49:24.294803 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.294752    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl"
Sep 09 21:49:24.294933 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.294812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77jrwl" event={"ID":"c447ac3e-70a6-4da4-9bdb-df938a706654","Type":"ContainerDied","Data":"c665b6c56be056cc74fc69b44715c318fa989ff6ee4cb4017bd09a7a53e53c34"}
Sep 09 21:49:24.294933 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.294855    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c665b6c56be056cc74fc69b44715c318fa989ff6ee4cb4017bd09a7a53e53c34"
Sep 09 21:49:24.297012 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.296996    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h"
Sep 09 21:49:24.297106 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.297028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57k2h" event={"ID":"5f58c891-a8f7-43ac-a83e-fd9fa08e2a30","Type":"ContainerDied","Data":"57b7996f8845c50404b2047fdac423b53e3ca0983830b7b1c711e8a8ebea8971"}
Sep 09 21:49:24.297106 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.297061    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57b7996f8845c50404b2047fdac423b53e3ca0983830b7b1c711e8a8ebea8971"
Sep 09 21:49:24.429891 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.429865    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:24.612029 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.611952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-util\") pod \"55f65f76-81da-43d4-823d-b0c8d96fa262\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") "
Sep 09 21:49:24.612171 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.612055    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt2bz\" (UniqueName: \"kubernetes.io/projected/55f65f76-81da-43d4-823d-b0c8d96fa262-kube-api-access-qt2bz\") pod \"55f65f76-81da-43d4-823d-b0c8d96fa262\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") "
Sep 09 21:49:24.612171 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.612086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-bundle\") pod \"55f65f76-81da-43d4-823d-b0c8d96fa262\" (UID: \"55f65f76-81da-43d4-823d-b0c8d96fa262\") "
Sep 09 21:49:24.612935 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.612902    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-bundle" (OuterVolumeSpecName: "bundle") pod "55f65f76-81da-43d4-823d-b0c8d96fa262" (UID: "55f65f76-81da-43d4-823d-b0c8d96fa262"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:24.614339 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.614318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f65f76-81da-43d4-823d-b0c8d96fa262-kube-api-access-qt2bz" (OuterVolumeSpecName: "kube-api-access-qt2bz") pod "55f65f76-81da-43d4-823d-b0c8d96fa262" (UID: "55f65f76-81da-43d4-823d-b0c8d96fa262"). InnerVolumeSpecName "kube-api-access-qt2bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:49:24.617531 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.617497    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-util" (OuterVolumeSpecName: "util") pod "55f65f76-81da-43d4-823d-b0c8d96fa262" (UID: "55f65f76-81da-43d4-823d-b0c8d96fa262"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:24.713457 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.713421    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:24.713457 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.713448    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55f65f76-81da-43d4-823d-b0c8d96fa262-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:24.713457 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:24.713457    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt2bz\" (UniqueName: \"kubernetes.io/projected/55f65f76-81da-43d4-823d-b0c8d96fa262-kube-api-access-qt2bz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:49:25.304093 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:25.304063    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh"
Sep 09 21:49:25.304093 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:25.304089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqbsnh" event={"ID":"55f65f76-81da-43d4-823d-b0c8d96fa262","Type":"ContainerDied","Data":"fd2f37a4bf1a8f4598b732d24212834c877b9e94b8a5aa776cb9746fffb9de7c"}
Sep 09 21:49:25.304609 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:25.304132    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd2f37a4bf1a8f4598b732d24212834c877b9e94b8a5aa776cb9746fffb9de7c"
Sep 09 21:49:26.476996 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:26.476964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:27.272549 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:27.272524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:30.499685 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:30.499655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:30.766405 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:30.766321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:32.220150 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:32.220115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:32.676514 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:32.676487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:33.093630 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:33.093599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:33.532936 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:33.532904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:33.814137 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:33.814052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:36.889425 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:36.889382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:38.319623 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:38.319587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:39.892611 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:39.892578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:39.989669 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:39.989638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:40.279919 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:40.279885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:40.381258 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:40.381209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:43.518657 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:43.518622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:43.625002 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:43.624955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:43.893009 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:43.892911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:43.994117 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:43.994090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:49.479609 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:49.479578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:49.577733 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:49.577704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:49.673276 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:49.673245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:56.594177 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:56.594129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:56.693948 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:56.693918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:57.735265 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:57.735237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:57.828468 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:49:57.828437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:00.365817 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:00.365779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:00.466006 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:00.465971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:00.958363 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:00.958334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:01.079210 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:01.079172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:03.383107 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:03.383071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:03.684455 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:03.684367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:09.354939 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:09.354899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:09.455007 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:09.454975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:09.555019 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:09.554989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:16.388061 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:16.388026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:16.522644 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:16.522584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:22.697590 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:22.697550    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:23.080233 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:23.080190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:24.768732 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.768690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:24.769554 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769535    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9p2f"]
Sep 09 21:50:24.769913 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769900    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77428fbf-b127-43d5-97e6-66e76a7c7ea5" containerName="pull"
Sep 09 21:50:24.769964 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769914    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="77428fbf-b127-43d5-97e6-66e76a7c7ea5" containerName="pull"
Sep 09 21:50:24.769964 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769927    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f65f76-81da-43d4-823d-b0c8d96fa262" containerName="pull"
Sep 09 21:50:24.769964 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769933    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f65f76-81da-43d4-823d-b0c8d96fa262" containerName="pull"
Sep 09 21:50:24.769964 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769946    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" containerName="util"
Sep 09 21:50:24.769964 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769952    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" containerName="util"
Sep 09 21:50:24.769964 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769958    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c447ac3e-70a6-4da4-9bdb-df938a706654" containerName="util"
Sep 09 21:50:24.769964 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769963    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c447ac3e-70a6-4da4-9bdb-df938a706654" containerName="util"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769968    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f65f76-81da-43d4-823d-b0c8d96fa262" containerName="util"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769973    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f65f76-81da-43d4-823d-b0c8d96fa262" containerName="util"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769981    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f65f76-81da-43d4-823d-b0c8d96fa262" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769986    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f65f76-81da-43d4-823d-b0c8d96fa262" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769993    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" containerName="pull"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.769998    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" containerName="pull"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770009    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77428fbf-b127-43d5-97e6-66e76a7c7ea5" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770013    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="77428fbf-b127-43d5-97e6-66e76a7c7ea5" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770020    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c447ac3e-70a6-4da4-9bdb-df938a706654" containerName="pull"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770025    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c447ac3e-70a6-4da4-9bdb-df938a706654" containerName="pull"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770030    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c447ac3e-70a6-4da4-9bdb-df938a706654" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770035    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c447ac3e-70a6-4da4-9bdb-df938a706654" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770040    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770044    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770051    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77428fbf-b127-43d5-97e6-66e76a7c7ea5" containerName="util"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770056    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="77428fbf-b127-43d5-97e6-66e76a7c7ea5" containerName="util"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770113    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f65f76-81da-43d4-823d-b0c8d96fa262" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770120    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="77428fbf-b127-43d5-97e6-66e76a7c7ea5" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770125    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f58c891-a8f7-43ac-a83e-fd9fa08e2a30" containerName="extract"
Sep 09 21:50:24.770206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.770133    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c447ac3e-70a6-4da4-9bdb-df938a706654" containerName="extract"
Sep 09 21:50:24.773074 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.773060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:24.775420 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.775401    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-252jf\""
Sep 09 21:50:24.775515 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.775401    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 21:50:24.775515 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.775437    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 21:50:24.776670 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.776655    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 21:50:24.780480 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.780462    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9p2f"]
Sep 09 21:50:24.871494 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.871458    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9p2f"]
Sep 09 21:50:24.877620 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.877593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:24.925979 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.925940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2lx\" (UniqueName: \"kubernetes.io/projected/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-kube-api-access-tj2lx\") pod \"limitador-limitador-8588749977-l9p2f\" (UID: \"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4\") " pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:24.926159 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.925994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-config-file\") pod \"limitador-limitador-8588749977-l9p2f\" (UID: \"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4\") " pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:24.933566 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:24.933538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:25.026913 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.026826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2lx\" (UniqueName: \"kubernetes.io/projected/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-kube-api-access-tj2lx\") pod \"limitador-limitador-8588749977-l9p2f\" (UID: \"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4\") " pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:25.026913 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.026900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-config-file\") pod \"limitador-limitador-8588749977-l9p2f\" (UID: \"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4\") " pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:25.027518 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.027498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-config-file\") pod \"limitador-limitador-8588749977-l9p2f\" (UID: \"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4\") " pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:25.034885 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.034855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2lx\" (UniqueName: \"kubernetes.io/projected/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-kube-api-access-tj2lx\") pod \"limitador-limitador-8588749977-l9p2f\" (UID: \"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4\") " pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:25.083596 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.083560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:25.356967 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.356941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9p2f"]
Sep 09 21:50:25.359498 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:50:25.359472    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ef5d66_71bb_4dd5_84d7_6d9d3ea760a4.slice/crio-cfc930c55728ea6f75f4832786edf8ff9ed740d10721216c42a4035d064fed2b WatchSource:0}: Error finding container cfc930c55728ea6f75f4832786edf8ff9ed740d10721216c42a4035d064fed2b: Status 404 returned error can't find the container with id cfc930c55728ea6f75f4832786edf8ff9ed740d10721216c42a4035d064fed2b
Sep 09 21:50:25.586715 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.586669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:25.620731 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.620642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f" event={"ID":"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4","Type":"ContainerStarted","Data":"cfc930c55728ea6f75f4832786edf8ff9ed740d10721216c42a4035d064fed2b"}
Sep 09 21:50:25.682822 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.682756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:25.743151 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.743120    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-mm9lv"]
Sep 09 21:50:25.748010 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.747986    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mm9lv"]
Sep 09 21:50:25.748141 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.748063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mm9lv"
Sep 09 21:50:25.750687 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.750664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kf294\""
Sep 09 21:50:25.935226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:25.935135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb98q\" (UniqueName: \"kubernetes.io/projected/8f430086-d5e3-44ca-8943-274968827140-kube-api-access-wb98q\") pod \"authorino-59c974bd48-mm9lv\" (UID: \"8f430086-d5e3-44ca-8943-274968827140\") " pod="kuadrant-system/authorino-59c974bd48-mm9lv"
Sep 09 21:50:26.036506 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:26.036450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb98q\" (UniqueName: \"kubernetes.io/projected/8f430086-d5e3-44ca-8943-274968827140-kube-api-access-wb98q\") pod \"authorino-59c974bd48-mm9lv\" (UID: \"8f430086-d5e3-44ca-8943-274968827140\") " pod="kuadrant-system/authorino-59c974bd48-mm9lv"
Sep 09 21:50:26.045738 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:26.045710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb98q\" (UniqueName: \"kubernetes.io/projected/8f430086-d5e3-44ca-8943-274968827140-kube-api-access-wb98q\") pod \"authorino-59c974bd48-mm9lv\" (UID: \"8f430086-d5e3-44ca-8943-274968827140\") " pod="kuadrant-system/authorino-59c974bd48-mm9lv"
Sep 09 21:50:26.058487 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:26.058457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mm9lv"
Sep 09 21:50:26.328318 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:26.328291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mm9lv"]
Sep 09 21:50:26.329671 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:50:26.329644    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f430086_d5e3_44ca_8943_274968827140.slice/crio-e89549a764ba89a66c976afac6617ae55b0cba0b8255da82c3eec5f2d1663a8d WatchSource:0}: Error finding container e89549a764ba89a66c976afac6617ae55b0cba0b8255da82c3eec5f2d1663a8d: Status 404 returned error can't find the container with id e89549a764ba89a66c976afac6617ae55b0cba0b8255da82c3eec5f2d1663a8d
Sep 09 21:50:26.629525 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:26.629427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mm9lv" event={"ID":"8f430086-d5e3-44ca-8943-274968827140","Type":"ContainerStarted","Data":"e89549a764ba89a66c976afac6617ae55b0cba0b8255da82c3eec5f2d1663a8d"}
Sep 09 21:50:28.643600 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:28.643560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f" event={"ID":"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4","Type":"ContainerStarted","Data":"31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529"}
Sep 09 21:50:28.644209 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:28.643719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:28.672637 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:28.672585    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f" podStartSLOduration=2.090685537 podStartE2EDuration="4.672569076s" totalImagesPullingTime="2.581883539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:28.672485468 +0000 UTC m=+721.569151475" watchObservedRunningTime="2026-09-09 21:50:28.672569076 +0000 UTC m=+721.569235084"
Sep 09 21:50:29.656586 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:29.656541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mm9lv" event={"ID":"8f430086-d5e3-44ca-8943-274968827140","Type":"ContainerStarted","Data":"163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b"}
Sep 09 21:50:29.671890 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:29.671835    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-mm9lv" podStartSLOduration=2.295125112 podStartE2EDuration="4.671820147s" totalImagesPullingTime="2.376695035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:29.670487173 +0000 UTC m=+722.567153182" watchObservedRunningTime="2026-09-09 21:50:29.671820147 +0000 UTC m=+722.568486155"
Sep 09 21:50:38.564674 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:38.564640    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9p2f"]
Sep 09 21:50:38.565156 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:38.564898    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f" podUID="e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4" containerName="limitador" containerID="cri-o://31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529" gracePeriod=30
Sep 09 21:50:38.565554 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:38.565501    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:39.105040 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.105010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:39.141692 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.141614    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-config-file\") pod \"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4\" (UID: \"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4\") "
Sep 09 21:50:39.141869 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.141692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj2lx\" (UniqueName: \"kubernetes.io/projected/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-kube-api-access-tj2lx\") pod \"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4\" (UID: \"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4\") "
Sep 09 21:50:39.142033 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.142010    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-config-file" (OuterVolumeSpecName: "config-file") pod "e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4" (UID: "e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:50:39.144103 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.144076    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-kube-api-access-tj2lx" (OuterVolumeSpecName: "kube-api-access-tj2lx") pod "e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4" (UID: "e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4"). InnerVolumeSpecName "kube-api-access-tj2lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:50:39.242235 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.242201    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj2lx\" (UniqueName: \"kubernetes.io/projected/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-kube-api-access-tj2lx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:50:39.242235 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.242232    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4-config-file\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:50:39.712141 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.712102    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4" containerID="31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529" exitCode=0
Sep 09 21:50:39.712565 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.712149    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f"
Sep 09 21:50:39.712565 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.712141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f" event={"ID":"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4","Type":"ContainerDied","Data":"31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529"}
Sep 09 21:50:39.712565 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.712260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l9p2f" event={"ID":"e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4","Type":"ContainerDied","Data":"cfc930c55728ea6f75f4832786edf8ff9ed740d10721216c42a4035d064fed2b"}
Sep 09 21:50:39.712565 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.712283    2563 scope.go:117] "RemoveContainer" containerID="31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529"
Sep 09 21:50:39.720876 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.720856    2563 scope.go:117] "RemoveContainer" containerID="31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529"
Sep 09 21:50:39.721127 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:50:39.721108    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529\": container with ID starting with 31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529 not found: ID does not exist" containerID="31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529"
Sep 09 21:50:39.721179 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.721136    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529"} err="failed to get container status \"31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529\": rpc error: code = NotFound desc = could not find container \"31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529\": container with ID starting with 31df58ef6132e59d20fb81d3c222f213ee02e5be56ef3cd64880125af680e529 not found: ID does not exist"
Sep 09 21:50:39.730348 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.730327    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9p2f"]
Sep 09 21:50:39.733785 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:39.733753    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9p2f"]
Sep 09 21:50:40.849538 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:40.849504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:40.950105 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:40.950071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:41.339369 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:41.339343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:41.611840 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:41.607426    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4" path="/var/lib/kubelet/pods/e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4/volumes"
Sep 09 21:50:51.248374 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.248338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:51.377031 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.376989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:51.717859 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.717783    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c4cf5dfb-xn4f8"]
Sep 09 21:50:51.718168 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.718156    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4" containerName="limitador"
Sep 09 21:50:51.718224 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.718169    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4" containerName="limitador"
Sep 09 21:50:51.718263 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.718242    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ef5d66-71bb-4dd5-84d7-6d9d3ea760a4" containerName="limitador"
Sep 09 21:50:51.721180 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.721165    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8"
Sep 09 21:50:51.723610 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.723591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 21:50:51.728811 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.728789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c4cf5dfb-xn4f8"]
Sep 09 21:50:51.738597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.738576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35e23bcb-1c66-41dc-832f-580adf76db5c-tls-cert\") pod \"authorino-5c4cf5dfb-xn4f8\" (UID: \"35e23bcb-1c66-41dc-832f-580adf76db5c\") " pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8"
Sep 09 21:50:51.738702 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.738621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgqv7\" (UniqueName: \"kubernetes.io/projected/35e23bcb-1c66-41dc-832f-580adf76db5c-kube-api-access-dgqv7\") pod \"authorino-5c4cf5dfb-xn4f8\" (UID: \"35e23bcb-1c66-41dc-832f-580adf76db5c\") " pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8"
Sep 09 21:50:51.839286 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.839252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgqv7\" (UniqueName: \"kubernetes.io/projected/35e23bcb-1c66-41dc-832f-580adf76db5c-kube-api-access-dgqv7\") pod \"authorino-5c4cf5dfb-xn4f8\" (UID: \"35e23bcb-1c66-41dc-832f-580adf76db5c\") " pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8"
Sep 09 21:50:51.839480 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.839347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35e23bcb-1c66-41dc-832f-580adf76db5c-tls-cert\") pod \"authorino-5c4cf5dfb-xn4f8\" (UID: \"35e23bcb-1c66-41dc-832f-580adf76db5c\") " pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8"
Sep 09 21:50:51.841785 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.841749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35e23bcb-1c66-41dc-832f-580adf76db5c-tls-cert\") pod \"authorino-5c4cf5dfb-xn4f8\" (UID: \"35e23bcb-1c66-41dc-832f-580adf76db5c\") " pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8"
Sep 09 21:50:51.846939 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:51.846915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgqv7\" (UniqueName: \"kubernetes.io/projected/35e23bcb-1c66-41dc-832f-580adf76db5c-kube-api-access-dgqv7\") pod \"authorino-5c4cf5dfb-xn4f8\" (UID: \"35e23bcb-1c66-41dc-832f-580adf76db5c\") " pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8"
Sep 09 21:50:52.031537 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:52.031502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8"
Sep 09 21:50:52.305115 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:52.305089    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c4cf5dfb-xn4f8"]
Sep 09 21:50:52.306840 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:50:52.306812    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35e23bcb_1c66_41dc_832f_580adf76db5c.slice/crio-60e4ea637469651fc95e16042349f40d485644e059fd1188a28459055943959e WatchSource:0}: Error finding container 60e4ea637469651fc95e16042349f40d485644e059fd1188a28459055943959e: Status 404 returned error can't find the container with id 60e4ea637469651fc95e16042349f40d485644e059fd1188a28459055943959e
Sep 09 21:50:52.782655 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:52.782631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8" event={"ID":"35e23bcb-1c66-41dc-832f-580adf76db5c","Type":"ContainerStarted","Data":"60e4ea637469651fc95e16042349f40d485644e059fd1188a28459055943959e"}
Sep 09 21:50:53.789657 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:53.789625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8" event={"ID":"35e23bcb-1c66-41dc-832f-580adf76db5c","Type":"ContainerStarted","Data":"a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8"}
Sep 09 21:50:53.804784 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:53.804721    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8" podStartSLOduration=2.342603361 podStartE2EDuration="2.804704307s" totalImagesPullingTime="462.100946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:53.802817222 +0000 UTC m=+746.699483227" watchObservedRunningTime="2026-09-09 21:50:53.804704307 +0000 UTC m=+746.701370314"
Sep 09 21:50:53.829405 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:53.829365    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mm9lv"]
Sep 09 21:50:53.829593 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:53.829569    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-mm9lv" podUID="8f430086-d5e3-44ca-8943-274968827140" containerName="authorino" containerID="cri-o://163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b" gracePeriod=30
Sep 09 21:50:54.076644 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.076619    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mm9lv"
Sep 09 21:50:54.164083 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.164046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb98q\" (UniqueName: \"kubernetes.io/projected/8f430086-d5e3-44ca-8943-274968827140-kube-api-access-wb98q\") pod \"8f430086-d5e3-44ca-8943-274968827140\" (UID: \"8f430086-d5e3-44ca-8943-274968827140\") "
Sep 09 21:50:54.166271 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.166242    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f430086-d5e3-44ca-8943-274968827140-kube-api-access-wb98q" (OuterVolumeSpecName: "kube-api-access-wb98q") pod "8f430086-d5e3-44ca-8943-274968827140" (UID: "8f430086-d5e3-44ca-8943-274968827140"). InnerVolumeSpecName "kube-api-access-wb98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:50:54.265487 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.265457    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb98q\" (UniqueName: \"kubernetes.io/projected/8f430086-d5e3-44ca-8943-274968827140-kube-api-access-wb98q\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:50:54.796208 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.796173    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f430086-d5e3-44ca-8943-274968827140" containerID="163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b" exitCode=0
Sep 09 21:50:54.796701 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.796218    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mm9lv"
Sep 09 21:50:54.796701 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.796260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mm9lv" event={"ID":"8f430086-d5e3-44ca-8943-274968827140","Type":"ContainerDied","Data":"163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b"}
Sep 09 21:50:54.796701 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.796303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mm9lv" event={"ID":"8f430086-d5e3-44ca-8943-274968827140","Type":"ContainerDied","Data":"e89549a764ba89a66c976afac6617ae55b0cba0b8255da82c3eec5f2d1663a8d"}
Sep 09 21:50:54.796701 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.796322    2563 scope.go:117] "RemoveContainer" containerID="163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b"
Sep 09 21:50:54.805573 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.805541    2563 scope.go:117] "RemoveContainer" containerID="163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b"
Sep 09 21:50:54.805874 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:50:54.805857    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b\": container with ID starting with 163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b not found: ID does not exist" containerID="163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b"
Sep 09 21:50:54.805984 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.805885    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b"} err="failed to get container status \"163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b\": rpc error: code = NotFound desc = could not find container \"163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b\": container with ID starting with 163fbbcbdb58aeef1691701f1f4d37db47bba6349acbde602dd1e79739767e0b not found: ID does not exist"
Sep 09 21:50:54.817921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.817891    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mm9lv"]
Sep 09 21:50:54.820251 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:54.820231    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mm9lv"]
Sep 09 21:50:55.609618 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:55.609585    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f430086-d5e3-44ca-8943-274968827140" path="/var/lib/kubelet/pods/8f430086-d5e3-44ca-8943-274968827140/volumes"
Sep 09 21:50:58.417117 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.417081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:58.444734 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.444689    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f4f78d787-5qrsr"]
Sep 09 21:50:58.445082 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.445067    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f430086-d5e3-44ca-8943-274968827140" containerName="authorino"
Sep 09 21:50:58.445132 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.445088    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f430086-d5e3-44ca-8943-274968827140" containerName="authorino"
Sep 09 21:50:58.445185 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.445168    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f430086-d5e3-44ca-8943-274968827140" containerName="authorino"
Sep 09 21:50:58.449606 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.449583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:58.452364 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.452343    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 21:50:58.452502 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.452481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8wcbp\""
Sep 09 21:50:58.452584 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.452354    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 21:50:58.452683 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.452377    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 21:50:58.459022 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.458683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f4f78d787-5qrsr"]
Sep 09 21:50:58.505839 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.505811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-webhook-cert\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:58.506010 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.505857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bn8b\" (UniqueName: \"kubernetes.io/projected/b32fb811-be6f-46c6-8a2b-5429ea59968e-kube-api-access-8bn8b\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:58.506010 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.505955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-metrics-certs\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:58.517021 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.516986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:58.607052 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.607009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-metrics-certs\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:58.607250 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.607115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-webhook-cert\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:58.607250 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:50:58.607122    2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 09 21:50:58.607250 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.607150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn8b\" (UniqueName: \"kubernetes.io/projected/b32fb811-be6f-46c6-8a2b-5429ea59968e-kube-api-access-8bn8b\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:58.607250 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:50:58.607181    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-metrics-certs podName:b32fb811-be6f-46c6-8a2b-5429ea59968e nodeName:}" failed. No retries permitted until 2026-09-09 21:50:59.107162385 +0000 UTC m=+752.003828376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-metrics-certs") pod "maas-controller-f4f78d787-5qrsr" (UID: "b32fb811-be6f-46c6-8a2b-5429ea59968e") : secret "maas-controller-metrics-tls" not found
Sep 09 21:50:58.607475 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:50:58.607269    2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 09 21:50:58.607475 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:50:58.607327    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-webhook-cert podName:b32fb811-be6f-46c6-8a2b-5429ea59968e nodeName:}" failed. No retries permitted until 2026-09-09 21:50:59.10730967 +0000 UTC m=+752.003975661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-webhook-cert") pod "maas-controller-f4f78d787-5qrsr" (UID: "b32fb811-be6f-46c6-8a2b-5429ea59968e") : secret "maas-controller-webhook-cert" not found
Sep 09 21:50:58.607869 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.607852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:58.613894 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.613870    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c999c7bf-kj7nh"]
Sep 09 21:50:58.617804 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.617748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:58.620264 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.620243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bn8b\" (UniqueName: \"kubernetes.io/projected/b32fb811-be6f-46c6-8a2b-5429ea59968e-kube-api-access-8bn8b\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:58.625127 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.625108    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-kj7nh"]
Sep 09 21:50:58.696969 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.696894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:58.708108 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.708083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-webhook-cert\") pod \"maas-controller-5c999c7bf-kj7nh\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") " pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:58.708245 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.708197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdjn\" (UniqueName: \"kubernetes.io/projected/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-kube-api-access-vfdjn\") pod \"maas-controller-5c999c7bf-kj7nh\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") " pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:58.708245 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.708233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-metrics-certs\") pod \"maas-controller-5c999c7bf-kj7nh\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") " pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:58.798488 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.798457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:58.809532 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.809506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdjn\" (UniqueName: \"kubernetes.io/projected/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-kube-api-access-vfdjn\") pod \"maas-controller-5c999c7bf-kj7nh\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") " pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:58.809642 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.809549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-metrics-certs\") pod \"maas-controller-5c999c7bf-kj7nh\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") " pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:58.809642 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.809612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-webhook-cert\") pod \"maas-controller-5c999c7bf-kj7nh\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") " pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:58.812151 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.812132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-webhook-cert\") pod \"maas-controller-5c999c7bf-kj7nh\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") " pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:58.812229 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.812162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-metrics-certs\") pod \"maas-controller-5c999c7bf-kj7nh\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") " pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:58.818057 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.818036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdjn\" (UniqueName: \"kubernetes.io/projected/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-kube-api-access-vfdjn\") pod \"maas-controller-5c999c7bf-kj7nh\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") " pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:58.940329 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:58.940295    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:50:59.111923 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:59.111894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-webhook-cert\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:59.112083 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:59.111945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-metrics-certs\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:59.114512 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:59.114479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-metrics-certs\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:59.114642 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:59.114540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-webhook-cert\") pod \"maas-controller-f4f78d787-5qrsr\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") " pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:59.212588 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:59.212561    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-kj7nh"]
Sep 09 21:50:59.214293 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:50:59.214268    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb4fc34_ed19_4e4f_8523_978122c6fa9e.slice/crio-225a27d04a33c9d5fd086c1f937cd6192137764df3eeadfdd10809c2098a94c9 WatchSource:0}: Error finding container 225a27d04a33c9d5fd086c1f937cd6192137764df3eeadfdd10809c2098a94c9: Status 404 returned error can't find the container with id 225a27d04a33c9d5fd086c1f937cd6192137764df3eeadfdd10809c2098a94c9
Sep 09 21:50:59.362891 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:59.362851    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:50:59.634613 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:59.634585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f4f78d787-5qrsr"]
Sep 09 21:50:59.640250 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:50:59.640216    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32fb811_be6f_46c6_8a2b_5429ea59968e.slice/crio-782eb5625e8a513f8cc523d9b33aaabfb1a7f715de2d374534fbed5597219853 WatchSource:0}: Error finding container 782eb5625e8a513f8cc523d9b33aaabfb1a7f715de2d374534fbed5597219853: Status 404 returned error can't find the container with id 782eb5625e8a513f8cc523d9b33aaabfb1a7f715de2d374534fbed5597219853
Sep 09 21:50:59.826017 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:59.825976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f4f78d787-5qrsr" event={"ID":"b32fb811-be6f-46c6-8a2b-5429ea59968e","Type":"ContainerStarted","Data":"782eb5625e8a513f8cc523d9b33aaabfb1a7f715de2d374534fbed5597219853"}
Sep 09 21:50:59.827421 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:50:59.827388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c999c7bf-kj7nh" event={"ID":"0cb4fc34-ed19-4e4f-8523-978122c6fa9e","Type":"ContainerStarted","Data":"225a27d04a33c9d5fd086c1f937cd6192137764df3eeadfdd10809c2098a94c9"}
Sep 09 21:51:02.072290 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:02.072253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:02.222258 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:02.222166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:02.848119 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:02.848077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f4f78d787-5qrsr" event={"ID":"b32fb811-be6f-46c6-8a2b-5429ea59968e","Type":"ContainerStarted","Data":"4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b"}
Sep 09 21:51:02.848342 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:02.848291    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:51:02.849728 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:02.849705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c999c7bf-kj7nh" event={"ID":"0cb4fc34-ed19-4e4f-8523-978122c6fa9e","Type":"ContainerStarted","Data":"6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb"}
Sep 09 21:51:02.849839 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:02.849812    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:51:02.868483 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:02.868431    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f4f78d787-5qrsr" podStartSLOduration=2.610982523 podStartE2EDuration="4.868415933s" totalImagesPullingTime="2.25743341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:51:02.866169075 +0000 UTC m=+755.762835082" watchObservedRunningTime="2026-09-09 21:51:02.868415933 +0000 UTC m=+755.765081950"
Sep 09 21:51:02.887054 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:02.886997    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c999c7bf-kj7nh" podStartSLOduration=2.210786424 podStartE2EDuration="4.886981951s" totalImagesPullingTime="2.676195527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:51:02.884342774 +0000 UTC m=+755.781008793" watchObservedRunningTime="2026-09-09 21:51:02.886981951 +0000 UTC m=+755.783648004"
Sep 09 21:51:13.857350 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:13.857319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 21:51:13.857867 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:13.857387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:51:13.919558 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:13.919507    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f4f78d787-5qrsr"]
Sep 09 21:51:13.919746 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:13.919723    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f4f78d787-5qrsr" podUID="b32fb811-be6f-46c6-8a2b-5429ea59968e" containerName="manager" containerID="cri-o://4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b" gracePeriod=30
Sep 09 21:51:14.160179 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.160150    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:51:14.261300 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.261271    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bn8b\" (UniqueName: \"kubernetes.io/projected/b32fb811-be6f-46c6-8a2b-5429ea59968e-kube-api-access-8bn8b\") pod \"b32fb811-be6f-46c6-8a2b-5429ea59968e\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") "
Sep 09 21:51:14.261489 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.261320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-metrics-certs\") pod \"b32fb811-be6f-46c6-8a2b-5429ea59968e\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") "
Sep 09 21:51:14.261489 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.261361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-webhook-cert\") pod \"b32fb811-be6f-46c6-8a2b-5429ea59968e\" (UID: \"b32fb811-be6f-46c6-8a2b-5429ea59968e\") "
Sep 09 21:51:14.263882 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.263848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b32fb811-be6f-46c6-8a2b-5429ea59968e" (UID: "b32fb811-be6f-46c6-8a2b-5429ea59968e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:51:14.263882 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.263865    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32fb811-be6f-46c6-8a2b-5429ea59968e-kube-api-access-8bn8b" (OuterVolumeSpecName: "kube-api-access-8bn8b") pod "b32fb811-be6f-46c6-8a2b-5429ea59968e" (UID: "b32fb811-be6f-46c6-8a2b-5429ea59968e"). InnerVolumeSpecName "kube-api-access-8bn8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:51:14.264043 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.263925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b32fb811-be6f-46c6-8a2b-5429ea59968e" (UID: "b32fb811-be6f-46c6-8a2b-5429ea59968e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:51:14.362136 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.362092    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-webhook-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:51:14.362136 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.362130    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bn8b\" (UniqueName: \"kubernetes.io/projected/b32fb811-be6f-46c6-8a2b-5429ea59968e-kube-api-access-8bn8b\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:51:14.362136 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.362141    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32fb811-be6f-46c6-8a2b-5429ea59968e-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:51:14.917452 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.917421    2563 generic.go:356] "Generic (PLEG): container finished" podID="b32fb811-be6f-46c6-8a2b-5429ea59968e" containerID="4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b" exitCode=0
Sep 09 21:51:14.917903 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.917461    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f4f78d787-5qrsr"
Sep 09 21:51:14.917903 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.917511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f4f78d787-5qrsr" event={"ID":"b32fb811-be6f-46c6-8a2b-5429ea59968e","Type":"ContainerDied","Data":"4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b"}
Sep 09 21:51:14.917903 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.917564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f4f78d787-5qrsr" event={"ID":"b32fb811-be6f-46c6-8a2b-5429ea59968e","Type":"ContainerDied","Data":"782eb5625e8a513f8cc523d9b33aaabfb1a7f715de2d374534fbed5597219853"}
Sep 09 21:51:14.917903 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.917586    2563 scope.go:117] "RemoveContainer" containerID="4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b"
Sep 09 21:51:14.934148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.933966    2563 scope.go:117] "RemoveContainer" containerID="4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b"
Sep 09 21:51:14.934346 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:51:14.934327    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b\": container with ID starting with 4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b not found: ID does not exist" containerID="4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b"
Sep 09 21:51:14.934399 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.934353    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b"} err="failed to get container status \"4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b\": rpc error: code = NotFound desc = could not find container \"4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b\": container with ID starting with 4eb3321b272417bdeb4218beb51c1063dacdfd450723afb0612c104096dcfc6b not found: ID does not exist"
Sep 09 21:51:14.943955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.943933    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f4f78d787-5qrsr"]
Sep 09 21:51:14.947841 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:14.947807    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f4f78d787-5qrsr"]
Sep 09 21:51:15.605846 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:15.605814    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b32fb811-be6f-46c6-8a2b-5429ea59968e" path="/var/lib/kubelet/pods/b32fb811-be6f-46c6-8a2b-5429ea59968e/volumes"
Sep 09 21:51:27.790512 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:27.790483    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:27.839673 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:27.839594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:34.987721 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:34.987679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:35.094288 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:35.094259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:36.473973 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.473942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:36.575053 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.575019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:36.609188 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.609155    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"]
Sep 09 21:51:36.609529 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.609516    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b32fb811-be6f-46c6-8a2b-5429ea59968e" containerName="manager"
Sep 09 21:51:36.609587 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.609530    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b32fb811-be6f-46c6-8a2b-5429ea59968e" containerName="manager"
Sep 09 21:51:36.609622 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.609611    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b32fb811-be6f-46c6-8a2b-5429ea59968e" containerName="manager"
Sep 09 21:51:36.614578 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.614557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.617129 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.617105    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 21:51:36.617267 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.617168    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 21:51:36.618208 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.618185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-94w96\""
Sep 09 21:51:36.618208 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.618206    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 21:51:36.618380 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.618207    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:51:36.623364 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.623339    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"]
Sep 09 21:51:36.649413 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.649359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpbvk\" (UniqueName: \"kubernetes.io/projected/d06ed691-a797-4a06-b9be-e08cfda44752-kube-api-access-tpbvk\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.649638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.649421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-maas-api-tls\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.649638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.649593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06ed691-a797-4a06-b9be-e08cfda44752-openshift-service-ca\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.649638 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.649633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-metrics-certs\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.674994 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.674954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:36.751136 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.751105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06ed691-a797-4a06-b9be-e08cfda44752-openshift-service-ca\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.751310 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.751141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-metrics-certs\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.751310 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.751190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpbvk\" (UniqueName: \"kubernetes.io/projected/d06ed691-a797-4a06-b9be-e08cfda44752-kube-api-access-tpbvk\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.751310 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.751210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-maas-api-tls\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.751783 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.751743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06ed691-a797-4a06-b9be-e08cfda44752-openshift-service-ca\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.754001 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.753977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-maas-api-tls\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.754111 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.754018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-metrics-certs\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.760030 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.760007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpbvk\" (UniqueName: \"kubernetes.io/projected/d06ed691-a797-4a06-b9be-e08cfda44752-kube-api-access-tpbvk\") pod \"maas-api-77bd4d9968-j8f8r\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") " pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:36.791751 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.791723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:36.926761 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:36.926715    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:37.010653 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.010482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:37.114179 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.114145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:37.152698 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.152663    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"]
Sep 09 21:51:37.156925 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.156899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"
Sep 09 21:51:37.159540 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.159516    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 21:51:37.159940 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.159924    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7hwm4\""
Sep 09 21:51:37.165947 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.165918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"]
Sep 09 21:51:37.206399 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.206374    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"]
Sep 09 21:51:37.208329 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:51:37.208275    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06ed691_a797_4a06_b9be_e08cfda44752.slice/crio-ebcf50471a3d1f11f4280302fba4bbbc1924692e09cf1366d15561e099aa1cfe WatchSource:0}: Error finding container ebcf50471a3d1f11f4280302fba4bbbc1924692e09cf1366d15561e099aa1cfe: Status 404 returned error can't find the container with id ebcf50471a3d1f11f4280302fba4bbbc1924692e09cf1366d15561e099aa1cfe
Sep 09 21:51:37.255713 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.255676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b4be3d-5973-4de5-ad82-0582fb44a438-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-9pcwv\" (UID: \"d5b4be3d-5973-4de5-ad82-0582fb44a438\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"
Sep 09 21:51:37.255922 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.255723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkpqp\" (UniqueName: \"kubernetes.io/projected/d5b4be3d-5973-4de5-ad82-0582fb44a438-kube-api-access-bkpqp\") pod \"payload-processing-5b54b7fbb7-9pcwv\" (UID: \"d5b4be3d-5973-4de5-ad82-0582fb44a438\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"
Sep 09 21:51:37.357095 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.356997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b4be3d-5973-4de5-ad82-0582fb44a438-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-9pcwv\" (UID: \"d5b4be3d-5973-4de5-ad82-0582fb44a438\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"
Sep 09 21:51:37.357095 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.357040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkpqp\" (UniqueName: \"kubernetes.io/projected/d5b4be3d-5973-4de5-ad82-0582fb44a438-kube-api-access-bkpqp\") pod \"payload-processing-5b54b7fbb7-9pcwv\" (UID: \"d5b4be3d-5973-4de5-ad82-0582fb44a438\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"
Sep 09 21:51:37.357863 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.357836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b4be3d-5973-4de5-ad82-0582fb44a438-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-9pcwv\" (UID: \"d5b4be3d-5973-4de5-ad82-0582fb44a438\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"
Sep 09 21:51:37.364915 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.364891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkpqp\" (UniqueName: \"kubernetes.io/projected/d5b4be3d-5973-4de5-ad82-0582fb44a438-kube-api-access-bkpqp\") pod \"payload-processing-5b54b7fbb7-9pcwv\" (UID: \"d5b4be3d-5973-4de5-ad82-0582fb44a438\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"
Sep 09 21:51:37.470967 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.470933    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"
Sep 09 21:51:37.748602 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:37.748568    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"]
Sep 09 21:51:37.756864 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:51:37.756837    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b4be3d_5973_4de5_ad82_0582fb44a438.slice/crio-47b69ee973e8ddfc4399748dd7a295dcfd3586626c70f669fbce1f749a58d127 WatchSource:0}: Error finding container 47b69ee973e8ddfc4399748dd7a295dcfd3586626c70f669fbce1f749a58d127: Status 404 returned error can't find the container with id 47b69ee973e8ddfc4399748dd7a295dcfd3586626c70f669fbce1f749a58d127
Sep 09 21:51:38.044230 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:38.044144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv" event={"ID":"d5b4be3d-5973-4de5-ad82-0582fb44a438","Type":"ContainerStarted","Data":"47b69ee973e8ddfc4399748dd7a295dcfd3586626c70f669fbce1f749a58d127"}
Sep 09 21:51:38.045605 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:38.045565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r" event={"ID":"d06ed691-a797-4a06-b9be-e08cfda44752","Type":"ContainerStarted","Data":"ebcf50471a3d1f11f4280302fba4bbbc1924692e09cf1366d15561e099aa1cfe"}
Sep 09 21:51:40.059084 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:40.059046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv" event={"ID":"d5b4be3d-5973-4de5-ad82-0582fb44a438","Type":"ContainerStarted","Data":"66dba24a3201eb329718d27be03343de0c15944e52f3283c94bd82dbfd3b0557"}
Sep 09 21:51:40.059556 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:40.059113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"
Sep 09 21:51:40.060891 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:40.060852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r" event={"ID":"d06ed691-a797-4a06-b9be-e08cfda44752","Type":"ContainerStarted","Data":"5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8"}
Sep 09 21:51:40.061026 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:40.060919    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:40.076547 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:40.076497    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv" podStartSLOduration=1.130374725 podStartE2EDuration="3.076484045s" totalImagesPullingTime="1.94610932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:51:40.075171855 +0000 UTC m=+792.971837863" watchObservedRunningTime="2026-09-09 21:51:40.076484045 +0000 UTC m=+792.973150052"
Sep 09 21:51:40.090464 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:40.090349    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r" podStartSLOduration=1.6026129770000002 podStartE2EDuration="4.090332782s" totalImagesPullingTime="2.487719805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:51:40.0903043 +0000 UTC m=+792.986970308" watchObservedRunningTime="2026-09-09 21:51:40.090332782 +0000 UTC m=+792.986998796"
Sep 09 21:51:46.073387 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:46.073357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 21:51:47.098631 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:47.098584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:47.198173 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:47.198145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:51.070603 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:51.070526    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-9pcwv"
Sep 09 21:51:52.404138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:51:52.404097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:06.449762 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:06.449725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:21.101599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:21.101568    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:31.644585 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:31.644552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:34.335210 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:34.335176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:37.387072 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:37.387042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:49.963865 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:49.963833    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"]
Sep 09 21:52:49.967499 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:49.967484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:49.971417 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:49.971395    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 21:52:49.971547 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:49.971395    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5x6f2\""
Sep 09 21:52:49.971547 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:49.971399    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 21:52:49.971547 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:49.971400    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 21:52:49.977854 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:49.977835    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"]
Sep 09 21:52:50.065695 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.065661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:50.091211 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.091176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea97abd-538f-4db7-9fe7-c52cce8a345e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.091362 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.091232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.091362 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.091254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.091362 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.091278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.091486 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.091381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.091486 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.091420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7v7\" (UniqueName: \"kubernetes.io/projected/3ea97abd-538f-4db7-9fe7-c52cce8a345e-kube-api-access-rg7v7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.192152 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.192123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.192342 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.192162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7v7\" (UniqueName: \"kubernetes.io/projected/3ea97abd-538f-4db7-9fe7-c52cce8a345e-kube-api-access-rg7v7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.192342 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.192195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea97abd-538f-4db7-9fe7-c52cce8a345e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.192342 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.192233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.192342 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.192248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.192342 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.192262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.192630 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.192565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.192689 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.192668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.192689 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.192672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.194824 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.194800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea97abd-538f-4db7-9fe7-c52cce8a345e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.195019 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.195002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea97abd-538f-4db7-9fe7-c52cce8a345e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.201752 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.201725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7v7\" (UniqueName: \"kubernetes.io/projected/3ea97abd-538f-4db7-9fe7-c52cce8a345e-kube-api-access-rg7v7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df\" (UID: \"3ea97abd-538f-4db7-9fe7-c52cce8a345e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.278616 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.278580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:52:50.556733 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:50.556708    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"]
Sep 09 21:52:50.558256 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:52:50.558215    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea97abd_538f_4db7_9fe7_c52cce8a345e.slice/crio-afecad50570a08b55e0f700c2d77fd8ea20ffdd1ae6aa3aa536464964109487a WatchSource:0}: Error finding container afecad50570a08b55e0f700c2d77fd8ea20ffdd1ae6aa3aa536464964109487a: Status 404 returned error can't find the container with id afecad50570a08b55e0f700c2d77fd8ea20ffdd1ae6aa3aa536464964109487a
Sep 09 21:52:51.462404 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:51.462322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df" event={"ID":"3ea97abd-538f-4db7-9fe7-c52cce8a345e","Type":"ContainerStarted","Data":"afecad50570a08b55e0f700c2d77fd8ea20ffdd1ae6aa3aa536464964109487a"}
Sep 09 21:52:52.469278 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:52.469239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df" event={"ID":"3ea97abd-538f-4db7-9fe7-c52cce8a345e","Type":"ContainerStarted","Data":"d902b557c626101e29dff7ccb7121a941fb71a7face6e522cb228b86645bafb0"}
Sep 09 21:52:52.488786 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:52:52.488730    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df" podStartSLOduration=2.419551598 podStartE2EDuration="3.488715362s" totalImagesPullingTime="1.069163764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:52:52.487199335 +0000 UTC m=+865.383865346" watchObservedRunningTime="2026-09-09 21:52:52.488715362 +0000 UTC m=+865.385381370"
Sep 09 21:53:00.279393 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:53:00.279355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:53:00.279828 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:53:00.279444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:53:00.291801 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:53:00.291760    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:53:00.523015 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:53:00.522982    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df"
Sep 09 21:53:01.179431 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:53:01.179391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:03.319291 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:53:03.319255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:04.253610 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:53:04.253581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:27.569722 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:53:27.569695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:53:27.574116 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:53:27.574094    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:53:57.704695 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:53:57.704662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:13.535888 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:13.535858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:13.590108 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:13.590080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:19.062079 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.062043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55b7f69cd5-928jh"]
Sep 09 21:54:19.065530 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.065512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b7f69cd5-928jh"
Sep 09 21:54:19.072877 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.072667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55b7f69cd5-928jh"]
Sep 09 21:54:19.155296 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.155265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dde1498b-bf96-43df-b65a-6a6cd5080c00-tls-cert\") pod \"authorino-55b7f69cd5-928jh\" (UID: \"dde1498b-bf96-43df-b65a-6a6cd5080c00\") " pod="kuadrant-system/authorino-55b7f69cd5-928jh"
Sep 09 21:54:19.155481 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.155339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rx4\" (UniqueName: \"kubernetes.io/projected/dde1498b-bf96-43df-b65a-6a6cd5080c00-kube-api-access-h8rx4\") pod \"authorino-55b7f69cd5-928jh\" (UID: \"dde1498b-bf96-43df-b65a-6a6cd5080c00\") " pod="kuadrant-system/authorino-55b7f69cd5-928jh"
Sep 09 21:54:19.256182 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.256145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dde1498b-bf96-43df-b65a-6a6cd5080c00-tls-cert\") pod \"authorino-55b7f69cd5-928jh\" (UID: \"dde1498b-bf96-43df-b65a-6a6cd5080c00\") " pod="kuadrant-system/authorino-55b7f69cd5-928jh"
Sep 09 21:54:19.256360 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.256257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rx4\" (UniqueName: \"kubernetes.io/projected/dde1498b-bf96-43df-b65a-6a6cd5080c00-kube-api-access-h8rx4\") pod \"authorino-55b7f69cd5-928jh\" (UID: \"dde1498b-bf96-43df-b65a-6a6cd5080c00\") " pod="kuadrant-system/authorino-55b7f69cd5-928jh"
Sep 09 21:54:19.258853 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.258826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dde1498b-bf96-43df-b65a-6a6cd5080c00-tls-cert\") pod \"authorino-55b7f69cd5-928jh\" (UID: \"dde1498b-bf96-43df-b65a-6a6cd5080c00\") " pod="kuadrant-system/authorino-55b7f69cd5-928jh"
Sep 09 21:54:19.264102 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.264074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rx4\" (UniqueName: \"kubernetes.io/projected/dde1498b-bf96-43df-b65a-6a6cd5080c00-kube-api-access-h8rx4\") pod \"authorino-55b7f69cd5-928jh\" (UID: \"dde1498b-bf96-43df-b65a-6a6cd5080c00\") " pod="kuadrant-system/authorino-55b7f69cd5-928jh"
Sep 09 21:54:19.376118 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.376038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b7f69cd5-928jh"
Sep 09 21:54:19.652250 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.652223    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55b7f69cd5-928jh"]
Sep 09 21:54:19.654055 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:54:19.654014    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddde1498b_bf96_43df_b65a_6a6cd5080c00.slice/crio-6552f6ae746c6ec5e555aa903b9c13f042997320397c6430381bbfdf20a5b302 WatchSource:0}: Error finding container 6552f6ae746c6ec5e555aa903b9c13f042997320397c6430381bbfdf20a5b302: Status 404 returned error can't find the container with id 6552f6ae746c6ec5e555aa903b9c13f042997320397c6430381bbfdf20a5b302
Sep 09 21:54:19.655370 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.655354    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:54:19.949734 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:19.949651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b7f69cd5-928jh" event={"ID":"dde1498b-bf96-43df-b65a-6a6cd5080c00","Type":"ContainerStarted","Data":"6552f6ae746c6ec5e555aa903b9c13f042997320397c6430381bbfdf20a5b302"}
Sep 09 21:54:20.958140 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:20.958094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b7f69cd5-928jh" event={"ID":"dde1498b-bf96-43df-b65a-6a6cd5080c00","Type":"ContainerStarted","Data":"023f52d621e635678bd1b055e17b86d8baf39c536cb5aad783f29cb623394752"}
Sep 09 21:54:20.976410 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:20.976359    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55b7f69cd5-928jh" podStartSLOduration=1.25005545 podStartE2EDuration="1.976344071s" totalImagesPullingTime="726.288621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:20.975176658 +0000 UTC m=+953.871842693" watchObservedRunningTime="2026-09-09 21:54:20.976344071 +0000 UTC m=+953.873010079"
Sep 09 21:54:21.011998 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.011963    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c4cf5dfb-xn4f8"]
Sep 09 21:54:21.012214 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.012190    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8" podUID="35e23bcb-1c66-41dc-832f-580adf76db5c" containerName="authorino" containerID="cri-o://a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8" gracePeriod=30
Sep 09 21:54:21.256739 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.256710    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8"
Sep 09 21:54:21.271245 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.271135    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgqv7\" (UniqueName: \"kubernetes.io/projected/35e23bcb-1c66-41dc-832f-580adf76db5c-kube-api-access-dgqv7\") pod \"35e23bcb-1c66-41dc-832f-580adf76db5c\" (UID: \"35e23bcb-1c66-41dc-832f-580adf76db5c\") "
Sep 09 21:54:21.271245 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.271178    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35e23bcb-1c66-41dc-832f-580adf76db5c-tls-cert\") pod \"35e23bcb-1c66-41dc-832f-580adf76db5c\" (UID: \"35e23bcb-1c66-41dc-832f-580adf76db5c\") "
Sep 09 21:54:21.274733 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.274599    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e23bcb-1c66-41dc-832f-580adf76db5c-kube-api-access-dgqv7" (OuterVolumeSpecName: "kube-api-access-dgqv7") pod "35e23bcb-1c66-41dc-832f-580adf76db5c" (UID: "35e23bcb-1c66-41dc-832f-580adf76db5c"). InnerVolumeSpecName "kube-api-access-dgqv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:54:21.294792 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.292046    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e23bcb-1c66-41dc-832f-580adf76db5c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "35e23bcb-1c66-41dc-832f-580adf76db5c" (UID: "35e23bcb-1c66-41dc-832f-580adf76db5c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:54:21.372149 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.372115    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgqv7\" (UniqueName: \"kubernetes.io/projected/35e23bcb-1c66-41dc-832f-580adf76db5c-kube-api-access-dgqv7\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:54:21.372149 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.372145    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35e23bcb-1c66-41dc-832f-580adf76db5c-tls-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:54:21.534892 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.534845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:21.614609 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.614583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:21.966962 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.966876    2563 generic.go:356] "Generic (PLEG): container finished" podID="35e23bcb-1c66-41dc-832f-580adf76db5c" containerID="a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8" exitCode=0
Sep 09 21:54:21.966962 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.966913    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8"
Sep 09 21:54:21.967448 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.966956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8" event={"ID":"35e23bcb-1c66-41dc-832f-580adf76db5c","Type":"ContainerDied","Data":"a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8"}
Sep 09 21:54:21.967448 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.967003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c4cf5dfb-xn4f8" event={"ID":"35e23bcb-1c66-41dc-832f-580adf76db5c","Type":"ContainerDied","Data":"60e4ea637469651fc95e16042349f40d485644e059fd1188a28459055943959e"}
Sep 09 21:54:21.967448 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.967026    2563 scope.go:117] "RemoveContainer" containerID="a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8"
Sep 09 21:54:21.976876 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.976854    2563 scope.go:117] "RemoveContainer" containerID="a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8"
Sep 09 21:54:21.977201 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:54:21.977181    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8\": container with ID starting with a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8 not found: ID does not exist" containerID="a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8"
Sep 09 21:54:21.977293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.977225    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8"} err="failed to get container status \"a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8\": rpc error: code = NotFound desc = could not find container \"a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8\": container with ID starting with a6fdda51423623d1711331760b7c126f33e673a0e642d59e48a7130d03d39ea8 not found: ID does not exist"
Sep 09 21:54:21.987353 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.987322    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c4cf5dfb-xn4f8"]
Sep 09 21:54:21.991866 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:21.991843    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c4cf5dfb-xn4f8"]
Sep 09 21:54:22.707704 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:22.707648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:22.800565 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:22.800530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:23.606536 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:23.606504    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35e23bcb-1c66-41dc-832f-580adf76db5c" path="/var/lib/kubelet/pods/35e23bcb-1c66-41dc-832f-580adf76db5c/volumes"
Sep 09 21:54:41.297041 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:41.297007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:41.412589 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:41.412552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:41.633672 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:41.633590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:41.911556 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:41.911481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:42.790309 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:42.790276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:42.884609 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:42.884578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:43.227117 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.227042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:43.312509 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.312465    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7773-139974919520048"]
Sep 09 21:54:43.312967 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.312948    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e23bcb-1c66-41dc-832f-580adf76db5c" containerName="authorino"
Sep 09 21:54:43.313073 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.312968    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e23bcb-1c66-41dc-832f-580adf76db5c" containerName="authorino"
Sep 09 21:54:43.313135 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.313091    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="35e23bcb-1c66-41dc-832f-580adf76db5c" containerName="authorino"
Sep 09 21:54:43.317635 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.317614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:43.320287 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.320263    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fkr9w\""
Sep 09 21:54:43.322737 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.322712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-139974919520048"]
Sep 09 21:54:43.363469 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.363441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txtml\" (UniqueName: \"kubernetes.io/projected/d4f8a53f-a665-4aac-904a-c615ba71f002-kube-api-access-txtml\") pod \"test-curl-7773-139974919520048\" (UID: \"d4f8a53f-a665-4aac-904a-c615ba71f002\") " pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:43.464458 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.464427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txtml\" (UniqueName: \"kubernetes.io/projected/d4f8a53f-a665-4aac-904a-c615ba71f002-kube-api-access-txtml\") pod \"test-curl-7773-139974919520048\" (UID: \"d4f8a53f-a665-4aac-904a-c615ba71f002\") " pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:43.483409 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.483337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txtml\" (UniqueName: \"kubernetes.io/projected/d4f8a53f-a665-4aac-904a-c615ba71f002-kube-api-access-txtml\") pod \"test-curl-7773-139974919520048\" (UID: \"d4f8a53f-a665-4aac-904a-c615ba71f002\") " pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:43.627691 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.627657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:43.903825 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:43.903799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-139974919520048"]
Sep 09 21:54:43.906297 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:54:43.906263    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4f8a53f_a665_4aac_904a_c615ba71f002.slice/crio-0e75fa92ef350d42f5b4ce773bb94190b034e929bd7c83b3cb223b05439c0c28 WatchSource:0}: Error finding container 0e75fa92ef350d42f5b4ce773bb94190b034e929bd7c83b3cb223b05439c0c28: Status 404 returned error can't find the container with id 0e75fa92ef350d42f5b4ce773bb94190b034e929bd7c83b3cb223b05439c0c28
Sep 09 21:54:44.092242 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:44.092207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-139974919520048" event={"ID":"d4f8a53f-a665-4aac-904a-c615ba71f002","Type":"ContainerStarted","Data":"0e75fa92ef350d42f5b4ce773bb94190b034e929bd7c83b3cb223b05439c0c28"}
Sep 09 21:54:45.080984 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:45.080935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:45.099327 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:45.099290    2563 generic.go:356] "Generic (PLEG): container finished" podID="d4f8a53f-a665-4aac-904a-c615ba71f002" containerID="411477aa62867bcc4aa415a330d37659125b96f76f8ce31945d89f03951ffb12" exitCode=0
Sep 09 21:54:45.099496 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:45.099331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-139974919520048" event={"ID":"d4f8a53f-a665-4aac-904a-c615ba71f002","Type":"ContainerDied","Data":"411477aa62867bcc4aa415a330d37659125b96f76f8ce31945d89f03951ffb12"}
Sep 09 21:54:45.173077 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:45.172994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:46.178732 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.178701    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"]
Sep 09 21:54:46.184941 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.183179    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.187761 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.187593    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5aaa80-serving-cert\""
Sep 09 21:54:46.196898 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.196871    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"]
Sep 09 21:54:46.219584 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.219555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:46.249962 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.249938    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:46.268368 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.268343    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7773-139974919520048_d4f8a53f-a665-4aac-904a-c615ba71f002/test-curl-7773-139974919520048/0.log"
Sep 09 21:54:46.280848 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.280816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:46.288046 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.288020    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7773-139974919520048"]
Sep 09 21:54:46.291665 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.291642    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7773-139974919520048"]
Sep 09 21:54:46.293377 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.293360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls74g\" (UniqueName: \"kubernetes.io/projected/1653c335-347e-442d-9191-5b650bfa4d46-kube-api-access-ls74g\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.293433 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.293389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-metrics-certs\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.293502 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.293480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-maas-api-tls\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.293550 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.293533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1653c335-347e-442d-9191-5b650bfa4d46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.394877 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.394748    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txtml\" (UniqueName: \"kubernetes.io/projected/d4f8a53f-a665-4aac-904a-c615ba71f002-kube-api-access-txtml\") pod \"d4f8a53f-a665-4aac-904a-c615ba71f002\" (UID: \"d4f8a53f-a665-4aac-904a-c615ba71f002\") "
Sep 09 21:54:46.395072 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.395010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls74g\" (UniqueName: \"kubernetes.io/projected/1653c335-347e-442d-9191-5b650bfa4d46-kube-api-access-ls74g\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.395072 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.395060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-metrics-certs\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.395269 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.395168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-maas-api-tls\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.395269 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.395206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1653c335-347e-442d-9191-5b650bfa4d46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.396224 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.396176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1653c335-347e-442d-9191-5b650bfa4d46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.397746 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.397681    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f8a53f-a665-4aac-904a-c615ba71f002-kube-api-access-txtml" (OuterVolumeSpecName: "kube-api-access-txtml") pod "d4f8a53f-a665-4aac-904a-c615ba71f002" (UID: "d4f8a53f-a665-4aac-904a-c615ba71f002"). InnerVolumeSpecName "kube-api-access-txtml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:54:46.398273 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.398028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-metrics-certs\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.398273 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.398175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-maas-api-tls\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.410892 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.410868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls74g\" (UniqueName: \"kubernetes.io/projected/1653c335-347e-442d-9191-5b650bfa4d46-kube-api-access-ls74g\") pod \"maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.411620 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.411596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:46.417552 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.417530    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7773-139974919520048"]
Sep 09 21:54:46.417959 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.417945    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f8a53f-a665-4aac-904a-c615ba71f002" containerName="test-curl-7773-139974919520048"
Sep 09 21:54:46.418009 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.417960    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f8a53f-a665-4aac-904a-c615ba71f002" containerName="test-curl-7773-139974919520048"
Sep 09 21:54:46.418045 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.418023    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f8a53f-a665-4aac-904a-c615ba71f002" containerName="test-curl-7773-139974919520048"
Sep 09 21:54:46.421267 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.421248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:46.427643 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.427576    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-139974919520048"]
Sep 09 21:54:46.496395 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.496363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfrkr\" (UniqueName: \"kubernetes.io/projected/c4526a9b-1e5b-4a95-a585-fd3b5ec20004-kube-api-access-xfrkr\") pod \"test-curl-7773-139974919520048\" (UID: \"c4526a9b-1e5b-4a95-a585-fd3b5ec20004\") " pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:46.496565 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.496423    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txtml\" (UniqueName: \"kubernetes.io/projected/d4f8a53f-a665-4aac-904a-c615ba71f002-kube-api-access-txtml\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:54:46.500267 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.500239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:46.597239 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.597207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfrkr\" (UniqueName: \"kubernetes.io/projected/c4526a9b-1e5b-4a95-a585-fd3b5ec20004-kube-api-access-xfrkr\") pod \"test-curl-7773-139974919520048\" (UID: \"c4526a9b-1e5b-4a95-a585-fd3b5ec20004\") " pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:46.605803 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.605761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfrkr\" (UniqueName: \"kubernetes.io/projected/c4526a9b-1e5b-4a95-a585-fd3b5ec20004-kube-api-access-xfrkr\") pod \"test-curl-7773-139974919520048\" (UID: \"c4526a9b-1e5b-4a95-a585-fd3b5ec20004\") " pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:46.731840 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.731739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:46.776942 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.776902    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"]
Sep 09 21:54:46.778670 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:54:46.778639    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1653c335_347e_442d_9191_5b650bfa4d46.slice/crio-fbfeb532517566a91b034f81274406ce9814d92bfd6582cb441d09140e6cbf31 WatchSource:0}: Error finding container fbfeb532517566a91b034f81274406ce9814d92bfd6582cb441d09140e6cbf31: Status 404 returned error can't find the container with id fbfeb532517566a91b034f81274406ce9814d92bfd6582cb441d09140e6cbf31
Sep 09 21:54:46.802279 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.802200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:46.901098 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:46.901070    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:47.014829 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:47.014801    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-139974919520048"]
Sep 09 21:54:47.016386 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:54:47.016364    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4526a9b_1e5b_4a95_a585_fd3b5ec20004.slice/crio-271a2c7817a1892d73119a40ef403f419cad6ce4ae76c8bee34b2cd279114bc4 WatchSource:0}: Error finding container 271a2c7817a1892d73119a40ef403f419cad6ce4ae76c8bee34b2cd279114bc4: Status 404 returned error can't find the container with id 271a2c7817a1892d73119a40ef403f419cad6ce4ae76c8bee34b2cd279114bc4
Sep 09 21:54:47.111995 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:47.111938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95" event={"ID":"1653c335-347e-442d-9191-5b650bfa4d46","Type":"ContainerStarted","Data":"fbfeb532517566a91b034f81274406ce9814d92bfd6582cb441d09140e6cbf31"}
Sep 09 21:54:47.113424 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:47.113401    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:47.113597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:47.113450    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e75fa92ef350d42f5b4ce773bb94190b034e929bd7c83b3cb223b05439c0c28"
Sep 09 21:54:47.114860 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:47.114836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-139974919520048" event={"ID":"c4526a9b-1e5b-4a95-a585-fd3b5ec20004","Type":"ContainerStarted","Data":"271a2c7817a1892d73119a40ef403f419cad6ce4ae76c8bee34b2cd279114bc4"}
Sep 09 21:54:47.125054 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:47.125014    2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7773-139974919520048" oldPodUID="d4f8a53f-a665-4aac-904a-c615ba71f002" podUID="c4526a9b-1e5b-4a95-a585-fd3b5ec20004"
Sep 09 21:54:47.607663 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:47.607584    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f8a53f-a665-4aac-904a-c615ba71f002" path="/var/lib/kubelet/pods/d4f8a53f-a665-4aac-904a-c615ba71f002/volumes"
Sep 09 21:54:48.122130 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:48.122089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95" event={"ID":"1653c335-347e-442d-9191-5b650bfa4d46","Type":"ContainerStarted","Data":"cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46"}
Sep 09 21:54:48.122375 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:48.122281    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:48.123722 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:48.123685    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4526a9b-1e5b-4a95-a585-fd3b5ec20004" containerID="a3103fd6ee17adebe874e13d2f0338637984ddff4c95e5a2a6151e5922668878" exitCode=0
Sep 09 21:54:48.123848 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:48.123755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-139974919520048" event={"ID":"c4526a9b-1e5b-4a95-a585-fd3b5ec20004","Type":"ContainerDied","Data":"a3103fd6ee17adebe874e13d2f0338637984ddff4c95e5a2a6151e5922668878"}
Sep 09 21:54:48.144332 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:48.144285    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95" podStartSLOduration=1.5829755300000001 podStartE2EDuration="2.144269702s" totalImagesPullingTime="561.294172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:48.143361761 +0000 UTC m=+981.040027770" watchObservedRunningTime="2026-09-09 21:54:48.144269702 +0000 UTC m=+981.040935710"
Sep 09 21:54:49.136416 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.136388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:49.236755 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.236719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:49.262469 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.262443    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:49.281876 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.281850    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7773-139974919520048_c4526a9b-1e5b-4a95-a585-fd3b5ec20004/test-curl-7773-139974919520048/0.log"
Sep 09 21:54:49.304493 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.304461    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7773-139974919520048"]
Sep 09 21:54:49.307795 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.307759    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7773-139974919520048"]
Sep 09 21:54:49.329665 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.329640    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfrkr\" (UniqueName: \"kubernetes.io/projected/c4526a9b-1e5b-4a95-a585-fd3b5ec20004-kube-api-access-xfrkr\") pod \"c4526a9b-1e5b-4a95-a585-fd3b5ec20004\" (UID: \"c4526a9b-1e5b-4a95-a585-fd3b5ec20004\") "
Sep 09 21:54:49.331894 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.331865    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4526a9b-1e5b-4a95-a585-fd3b5ec20004-kube-api-access-xfrkr" (OuterVolumeSpecName: "kube-api-access-xfrkr") pod "c4526a9b-1e5b-4a95-a585-fd3b5ec20004" (UID: "c4526a9b-1e5b-4a95-a585-fd3b5ec20004"). InnerVolumeSpecName "kube-api-access-xfrkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:54:49.358824 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.358798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:49.430921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.430831    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfrkr\" (UniqueName: \"kubernetes.io/projected/c4526a9b-1e5b-4a95-a585-fd3b5ec20004-kube-api-access-xfrkr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:54:49.608196 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.608163    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4526a9b-1e5b-4a95-a585-fd3b5ec20004" path="/var/lib/kubelet/pods/c4526a9b-1e5b-4a95-a585-fd3b5ec20004/volumes"
Sep 09 21:54:49.645628 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.645596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:49.810078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:49.810050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:50.075548 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.075455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:50.076718 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.076696    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"]
Sep 09 21:54:50.077300 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.077285    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4526a9b-1e5b-4a95-a585-fd3b5ec20004" containerName="test-curl-7773-139974919520048"
Sep 09 21:54:50.077343 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.077304    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4526a9b-1e5b-4a95-a585-fd3b5ec20004" containerName="test-curl-7773-139974919520048"
Sep 09 21:54:50.077436 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.077425    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4526a9b-1e5b-4a95-a585-fd3b5ec20004" containerName="test-curl-7773-139974919520048"
Sep 09 21:54:50.081143 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.081121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.083861 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.083756    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 21:54:50.083994 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.083834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-594ab9-openshift-default-dockercfg-bb77b\""
Sep 09 21:54:50.093661 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.093588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"]
Sep 09 21:54:50.136289 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.136251    2563 scope.go:117] "RemoveContainer" containerID="a3103fd6ee17adebe874e13d2f0338637984ddff4c95e5a2a6151e5922668878"
Sep 09 21:54:50.136461 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.136253    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-139974919520048"
Sep 09 21:54:50.149557 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.149529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:50.238138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.238098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-token\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.238323 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.238223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-envoy\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.238323 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.238274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45w6\" (UniqueName: \"kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-kube-api-access-h45w6\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.238425 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.238369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-socket\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.238425 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.238417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-credential-socket\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.238507 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.238452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-data\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.238507 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.238485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istiod-ca-cert\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.238615 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.238576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-certs\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.238671 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.238624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-podinfo\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.339963 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.339873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-token\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.340137 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.339966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-envoy\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.340137 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.340009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45w6\" (UniqueName: \"kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-kube-api-access-h45w6\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.340137 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.340041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-socket\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.340137 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.340078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-credential-socket\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.340137 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.340107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-data\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.340411 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.340142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istiod-ca-cert\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.340411 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.340201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-certs\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.340411 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.340251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-podinfo\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.340631 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.340604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-credential-socket\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.340920 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.340895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-socket\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.341014 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.340947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-certs\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.341226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.341208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-data\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.341307 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.341280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istiod-ca-cert\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.342902 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.342876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-envoy\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.343524 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.343507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-podinfo\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.351802 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.351710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45w6\" (UniqueName: \"kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-kube-api-access-h45w6\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.351802 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.351714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-token\") pod \"e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") " pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.396027 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.395984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:50.403338 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.403309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:50.680476 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:50.680448    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"]
Sep 09 21:54:50.683624 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:54:50.683585    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c59c6d_a4ef_4bd5_ba64_f2df6d176351.slice/crio-270a82946d83c5dfc1fdce74716f1b328a7579b37f4e245cac24d2647bdff4b2 WatchSource:0}: Error finding container 270a82946d83c5dfc1fdce74716f1b328a7579b37f4e245cac24d2647bdff4b2: Status 404 returned error can't find the container with id 270a82946d83c5dfc1fdce74716f1b328a7579b37f4e245cac24d2647bdff4b2
Sep 09 21:54:51.143524 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:51.143491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" event={"ID":"04c59c6d-a4ef-4bd5-ba64-f2df6d176351","Type":"ContainerStarted","Data":"270a82946d83c5dfc1fdce74716f1b328a7579b37f4e245cac24d2647bdff4b2"}
Sep 09 21:54:52.619146 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:52.619109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:52.718264 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:52.718224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:53.461956 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.461925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:53.562050 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.562018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:53.849994 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.849964    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"]
Sep 09 21:54:53.853565 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.853543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:53.856203 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.856182    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ff5f88f5-serving-cert\""
Sep 09 21:54:53.867934 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.867910    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"]
Sep 09 21:54:53.870128 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.870079    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 21:54:53.870248 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.870228    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 21:54:53.870334 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.870313    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 21:54:53.887601 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.887552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:53.979090 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.979052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-metrics-certs\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:53.979090 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.979093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k889\" (UniqueName: \"kubernetes.io/projected/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-kube-api-access-2k889\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:53.979307 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.979130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-maas-api-tls\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:53.979307 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.979153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-openshift-service-ca\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:53.988528 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:53.988498    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:54.080002 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.079964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-metrics-certs\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:54.080200 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.080006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k889\" (UniqueName: \"kubernetes.io/projected/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-kube-api-access-2k889\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:54.080200 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.080042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-maas-api-tls\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:54.080200 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.080066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-openshift-service-ca\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:54.080200 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:54:54.080160    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-serving-cert: secret "maas-api-e2e-ait-ff5f88f5-serving-cert" not found
Sep 09 21:54:54.080433 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:54:54.080229    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-maas-api-tls podName:5059685f-60a6-4168-bc7d-bec5ccf5ff4b nodeName:}" failed. No retries permitted until 2026-09-09 21:54:54.580208955 +0000 UTC m=+987.476874947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-maas-api-tls") pod "maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg" (UID: "5059685f-60a6-4168-bc7d-bec5ccf5ff4b") : secret "maas-api-e2e-ait-ff5f88f5-serving-cert" not found
Sep 09 21:54:54.080877 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.080854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-openshift-service-ca\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:54.082866 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.082836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-metrics-certs\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:54.088543 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.088517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k889\" (UniqueName: \"kubernetes.io/projected/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-kube-api-access-2k889\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:54.137158 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.137080    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:54:54.167242 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.167208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" event={"ID":"04c59c6d-a4ef-4bd5-ba64-f2df6d176351","Type":"ContainerStarted","Data":"b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd"}
Sep 09 21:54:54.190710 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.189528    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" podStartSLOduration=1.006010612 podStartE2EDuration="4.189510347s" totalImagesPullingTime="3.183499735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:54.189114073 +0000 UTC m=+987.085780117" watchObservedRunningTime="2026-09-09 21:54:54.189510347 +0000 UTC m=+987.086176355"
Sep 09 21:54:54.320875 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.320842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:54.397088 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.396992    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:54.398649 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.398620    2563 patch_prober.go:28] interesting pod/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 09 21:54:54.398796 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.398694    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 09 21:54:54.448630 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.448597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:54.551366 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.551337    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:54.584921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.584886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-maas-api-tls\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:54.587805 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.587752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-maas-api-tls\") pod \"maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:54.770458 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:54.770425    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:55.047593 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:55.047568    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"]
Sep 09 21:54:55.049371 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:54:55.049340    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5059685f_60a6_4168_bc7d_bec5ccf5ff4b.slice/crio-9c59e76021b8e7bfdf4bc4252667d7dfb69fa44a0feb7f78785389eb7ef2aaa8 WatchSource:0}: Error finding container 9c59e76021b8e7bfdf4bc4252667d7dfb69fa44a0feb7f78785389eb7ef2aaa8: Status 404 returned error can't find the container with id 9c59e76021b8e7bfdf4bc4252667d7dfb69fa44a0feb7f78785389eb7ef2aaa8
Sep 09 21:54:55.175001 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:55.174961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg" event={"ID":"5059685f-60a6-4168-bc7d-bec5ccf5ff4b","Type":"ContainerStarted","Data":"9c59e76021b8e7bfdf4bc4252667d7dfb69fa44a0feb7f78785389eb7ef2aaa8"}
Sep 09 21:54:55.401069 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:55.401045    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:56.183173 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:56.183135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg" event={"ID":"5059685f-60a6-4168-bc7d-bec5ccf5ff4b","Type":"ContainerStarted","Data":"b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059"}
Sep 09 21:54:56.183669 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:56.183457    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:54:56.183669 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:56.183496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:56.184502 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:56.184484    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:54:56.197350 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:56.197253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:56.203104 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:56.203058    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg" podStartSLOduration=2.85622652 podStartE2EDuration="3.20304282s" totalImagesPullingTime="346.8163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:56.201782881 +0000 UTC m=+989.098448903" watchObservedRunningTime="2026-09-09 21:54:56.20304282 +0000 UTC m=+989.099708835"
Sep 09 21:54:56.293590 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:56.293549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:57.369819 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:57.369739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:57.443188 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:57.443152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:57.635067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:57.634944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:58.039800 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.039738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:58.144138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.144103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:58.246427 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.246385    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"]
Sep 09 21:54:58.250113 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.250085    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:54:58.254861 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.254837    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-594ab9-dockercfg-zx9ds\""
Sep 09 21:54:58.255285 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.255261    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-594ab9\""
Sep 09 21:54:58.265185 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.265157    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"]
Sep 09 21:54:58.422843 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.422723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v\" (UID: \"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:54:58.422843 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.422811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5sj\" (UniqueName: \"kubernetes.io/projected/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-kube-api-access-cd5sj\") pod \"payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v\" (UID: \"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:54:58.523338 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.523306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v\" (UID: \"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:54:58.523513 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.523369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5sj\" (UniqueName: \"kubernetes.io/projected/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-kube-api-access-cd5sj\") pod \"payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v\" (UID: \"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:54:58.524088 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.524069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v\" (UID: \"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:54:58.536089 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.536057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5sj\" (UniqueName: \"kubernetes.io/projected/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-kube-api-access-cd5sj\") pod \"payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v\" (UID: \"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:54:58.560892 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.560862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:54:58.836101 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:58.836075    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"]
Sep 09 21:54:58.837865 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:54:58.837831    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255ed03c_02f4_44cc_9cf4_4d7f2eaed76c.slice/crio-1a108ab06df245dbff3fc084a3adae2f4332b604b26e950679d28315ed7e00e9 WatchSource:0}: Error finding container 1a108ab06df245dbff3fc084a3adae2f4332b604b26e950679d28315ed7e00e9: Status 404 returned error can't find the container with id 1a108ab06df245dbff3fc084a3adae2f4332b604b26e950679d28315ed7e00e9
Sep 09 21:54:59.203741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:59.203648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v" event={"ID":"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c","Type":"ContainerStarted","Data":"42bc95916c699bc744b539d2f28282e443a537487bdd05c3be2f3ba1bc615309"}
Sep 09 21:54:59.203741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:59.203715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v" event={"ID":"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c","Type":"ContainerStarted","Data":"1a108ab06df245dbff3fc084a3adae2f4332b604b26e950679d28315ed7e00e9"}
Sep 09 21:54:59.204025 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:59.203833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:54:59.224945 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:59.224887    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v" podStartSLOduration=1.224870776 podStartE2EDuration="1.224870776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:59.22200163 +0000 UTC m=+992.118667639" watchObservedRunningTime="2026-09-09 21:54:59.224870776 +0000 UTC m=+992.121536801"
Sep 09 21:54:59.828967 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:59.828936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:54:59.992261 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:54:59.992215    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:00.089010 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:00.088918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:00.187281 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:00.187248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:02.197656 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:02.197625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:55:02.455152 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:02.455072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:02.554961 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:02.554923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:02.678139 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:02.678104    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:02.901074 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:02.901044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:04.488637 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:04.488606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:08.306134 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:08.306102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:08.812059 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:08.812021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:08.916036 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:08.915989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:09.034617 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:09.034571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:09.277338 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:09.277307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:09.996861 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:09.996821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:10.211912 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:10.211881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:55:10.311361 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:10.311272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:10.664151 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:10.664072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:10.975547 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:10.975458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:15.015105 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:15.015069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:15.981338 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:15.981306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:16.080431 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:16.080402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:16.666204 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:16.666164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:16.791215 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:16.791187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:17.319569 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.319540    2563 generic.go:356] "Generic (PLEG): container finished" podID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerID="b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd" exitCode=0
Sep 09 21:55:17.319969 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.319613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" event={"ID":"04c59c6d-a4ef-4bd5-ba64-f2df6d176351","Type":"ContainerDied","Data":"b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd"}
Sep 09 21:55:17.320125 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.320106    2563 scope.go:117] "RemoveContainer" containerID="b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd"
Sep 09 21:55:17.321211 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.321157    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 21:55:17.321312 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.321227    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 21:55:17.321312 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.321262    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 21:55:17.794233 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.794202    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"]
Sep 09 21:55:17.797318 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.797290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:17.802957 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.802935    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ebe9b0-serving-cert\""
Sep 09 21:55:17.810913 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.810888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"]
Sep 09 21:55:17.825034 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.824991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:17.875879 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.875830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:17.917793 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.917094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-metrics-certs\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:17.917793 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.917184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272d003c-cb92-4bb9-88b0-b06a6d60dddb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:17.917793 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.917231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7qt\" (UniqueName: \"kubernetes.io/projected/272d003c-cb92-4bb9-88b0-b06a6d60dddb-kube-api-access-2z7qt\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:17.917793 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:17.917338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.024379 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.024336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.024677 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.024659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-metrics-certs\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.025656 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.025629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272d003c-cb92-4bb9-88b0-b06a6d60dddb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.025858 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.025836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272d003c-cb92-4bb9-88b0-b06a6d60dddb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.025943 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.025897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7qt\" (UniqueName: \"kubernetes.io/projected/272d003c-cb92-4bb9-88b0-b06a6d60dddb-kube-api-access-2z7qt\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.026081 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:55:18.026064    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-serving-cert: secret "maas-api-e2e-inf-b-ebe9b0-serving-cert" not found
Sep 09 21:55:18.026214 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:55:18.026203    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-maas-api-tls podName:272d003c-cb92-4bb9-88b0-b06a6d60dddb nodeName:}" failed. No retries permitted until 2026-09-09 21:55:18.526183951 +0000 UTC m=+1011.422849953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-maas-api-tls") pod "maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb" (UID: "272d003c-cb92-4bb9-88b0-b06a6d60dddb") : secret "maas-api-e2e-inf-b-ebe9b0-serving-cert" not found
Sep 09 21:55:18.029462 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.029434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-metrics-certs\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.034736 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.034707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7qt\" (UniqueName: \"kubernetes.io/projected/272d003c-cb92-4bb9-88b0-b06a6d60dddb-kube-api-access-2z7qt\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.211936 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.211840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:18.327228 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.327196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" event={"ID":"04c59c6d-a4ef-4bd5-ba64-f2df6d176351","Type":"ContainerStarted","Data":"83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229"}
Sep 09 21:55:18.412682 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.396927    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:55:18.412682 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.397048    2563 patch_prober.go:28] interesting pod/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 09 21:55:18.412682 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.397100    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 09 21:55:18.425845 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.425812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:18.531506 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.531471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.534243 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.534215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.535249 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.535228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:18.624660 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.624617    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"]
Sep 09 21:55:18.629111 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.629083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:55:18.631591 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.631570    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ebe9b0\""
Sep 09 21:55:18.631782 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.631600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ebe9b0-dockercfg-2nghk\""
Sep 09 21:55:18.637812 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.637788    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"]
Sep 09 21:55:18.718033 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.717986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:18.733604 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.733551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhr4\" (UniqueName: \"kubernetes.io/projected/abc200bc-76b9-4695-b496-c4d42ea37300-kube-api-access-mfhr4\") pod \"payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4\" (UID: \"abc200bc-76b9-4695-b496-c4d42ea37300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:55:18.733720 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.733667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc200bc-76b9-4695-b496-c4d42ea37300-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4\" (UID: \"abc200bc-76b9-4695-b496-c4d42ea37300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:55:18.834490 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.834401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc200bc-76b9-4695-b496-c4d42ea37300-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4\" (UID: \"abc200bc-76b9-4695-b496-c4d42ea37300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:55:18.834662 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.834507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhr4\" (UniqueName: \"kubernetes.io/projected/abc200bc-76b9-4695-b496-c4d42ea37300-kube-api-access-mfhr4\") pod \"payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4\" (UID: \"abc200bc-76b9-4695-b496-c4d42ea37300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:55:18.835130 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.835108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc200bc-76b9-4695-b496-c4d42ea37300-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4\" (UID: \"abc200bc-76b9-4695-b496-c4d42ea37300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:55:18.842856 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.842825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhr4\" (UniqueName: \"kubernetes.io/projected/abc200bc-76b9-4695-b496-c4d42ea37300-kube-api-access-mfhr4\") pod \"payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4\" (UID: \"abc200bc-76b9-4695-b496-c4d42ea37300\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:55:18.924409 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.924315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:18.953845 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:18.953808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:55:19.002049 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:19.002005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"]
Sep 09 21:55:19.003925 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:55:19.003890    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272d003c_cb92_4bb9_88b0_b06a6d60dddb.slice/crio-3b2aa63419cc02ac83fb1b6bdb3b44913683b6da7159a7ceb2ec67c98d007ead WatchSource:0}: Error finding container 3b2aa63419cc02ac83fb1b6bdb3b44913683b6da7159a7ceb2ec67c98d007ead: Status 404 returned error can't find the container with id 3b2aa63419cc02ac83fb1b6bdb3b44913683b6da7159a7ceb2ec67c98d007ead
Sep 09 21:55:19.249762 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:19.249736    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"]
Sep 09 21:55:19.251356 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:55:19.251318    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc200bc_76b9_4695_b496_c4d42ea37300.slice/crio-44e96f5cff458c6a8410e63983a706778a66558436677847dc79c71314fdeecd WatchSource:0}: Error finding container 44e96f5cff458c6a8410e63983a706778a66558436677847dc79c71314fdeecd: Status 404 returned error can't find the container with id 44e96f5cff458c6a8410e63983a706778a66558436677847dc79c71314fdeecd
Sep 09 21:55:19.335631 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:19.335597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4" event={"ID":"abc200bc-76b9-4695-b496-c4d42ea37300","Type":"ContainerStarted","Data":"44e96f5cff458c6a8410e63983a706778a66558436677847dc79c71314fdeecd"}
Sep 09 21:55:19.337157 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:19.337133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb" event={"ID":"272d003c-cb92-4bb9-88b0-b06a6d60dddb","Type":"ContainerStarted","Data":"3b2aa63419cc02ac83fb1b6bdb3b44913683b6da7159a7ceb2ec67c98d007ead"}
Sep 09 21:55:19.397475 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:19.397434    2563 patch_prober.go:28] interesting pod/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 09 21:55:19.397626 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:19.397496    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 09 21:55:19.710455 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:19.710331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:20.348017 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:20.347973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4" event={"ID":"abc200bc-76b9-4695-b496-c4d42ea37300","Type":"ContainerStarted","Data":"2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c"}
Sep 09 21:55:20.348535 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:20.348064    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:55:20.349750 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:20.349725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb" event={"ID":"272d003c-cb92-4bb9-88b0-b06a6d60dddb","Type":"ContainerStarted","Data":"0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b"}
Sep 09 21:55:20.349899 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:20.349808    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:20.363793 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:20.363699    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4" podStartSLOduration=2.363685188 podStartE2EDuration="2.363685188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:20.362723742 +0000 UTC m=+1013.259389788" watchObservedRunningTime="2026-09-09 21:55:20.363685188 +0000 UTC m=+1013.260351245"
Sep 09 21:55:20.385337 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:20.385294    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb" podStartSLOduration=3.044734043 podStartE2EDuration="3.385276715s" totalImagesPullingTime="340.542672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:20.382837552 +0000 UTC m=+1013.279503559" watchObservedRunningTime="2026-09-09 21:55:20.385276715 +0000 UTC m=+1013.281942736"
Sep 09 21:55:20.413804 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:20.406709    2563 patch_prober.go:28] interesting pod/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 09 21:55:20.413804 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:20.406789    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 09 21:55:20.507005 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:20.506964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:21.137304 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:21.137260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:21.297962 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:21.297926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:21.397492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:21.397390    2563 patch_prober.go:28] interesting pod/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 09 21:55:21.397492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:21.397455    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 09 21:55:22.397346 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:22.397298    2563 patch_prober.go:28] interesting pod/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 09 21:55:22.397572 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:22.397370    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 09 21:55:23.397208 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:23.397170    2563 patch_prober.go:28] interesting pod/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 09 21:55:23.397440 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:23.397226    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 09 21:55:24.400474 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:24.400445    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:55:24.400903 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:24.400711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:55:24.401571 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:24.401556    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:55:26.362246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:26.362216    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:55:31.358826 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:31.358790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:55:34.142903 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:34.142866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:34.790420 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:34.790386    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"]
Sep 09 21:55:34.790652 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:34.790629    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg" podUID="5059685f-60a6-4168-bc7d-bec5ccf5ff4b" containerName="maas-api" containerID="cri-o://b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059" gracePeriod=30
Sep 09 21:55:34.815759 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:34.815733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:34.893751 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:34.893680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:34.955013 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:34.954964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:35.019828 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.019695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:35.054181 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.054118    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:55:35.100577 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.100543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:35.165966 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.165938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:35.192149 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.192113    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-metrics-certs\") pod \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") "
Sep 09 21:55:35.192336 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.192171    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k889\" (UniqueName: \"kubernetes.io/projected/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-kube-api-access-2k889\") pod \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") "
Sep 09 21:55:35.192336 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.192244    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-maas-api-tls\") pod \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") "
Sep 09 21:55:35.192336 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.192308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-openshift-service-ca\") pod \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\" (UID: \"5059685f-60a6-4168-bc7d-bec5ccf5ff4b\") "
Sep 09 21:55:35.192727 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.192694    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5059685f-60a6-4168-bc7d-bec5ccf5ff4b" (UID: "5059685f-60a6-4168-bc7d-bec5ccf5ff4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:55:35.194824 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.194737    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5059685f-60a6-4168-bc7d-bec5ccf5ff4b" (UID: "5059685f-60a6-4168-bc7d-bec5ccf5ff4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:55:35.194824 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.194779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-kube-api-access-2k889" (OuterVolumeSpecName: "kube-api-access-2k889") pod "5059685f-60a6-4168-bc7d-bec5ccf5ff4b" (UID: "5059685f-60a6-4168-bc7d-bec5ccf5ff4b"). InnerVolumeSpecName "kube-api-access-2k889". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:55:35.194986 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.194879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5059685f-60a6-4168-bc7d-bec5ccf5ff4b" (UID: "5059685f-60a6-4168-bc7d-bec5ccf5ff4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:55:35.293310 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.293264    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:55:35.293310 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.293304    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:55:35.293310 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.293314    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:55:35.293310 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.293323    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k889\" (UniqueName: \"kubernetes.io/projected/5059685f-60a6-4168-bc7d-bec5ccf5ff4b-kube-api-access-2k889\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:55:35.441533 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.441432    2563 generic.go:356] "Generic (PLEG): container finished" podID="5059685f-60a6-4168-bc7d-bec5ccf5ff4b" containerID="b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059" exitCode=0
Sep 09 21:55:35.441533 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.441479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg" event={"ID":"5059685f-60a6-4168-bc7d-bec5ccf5ff4b","Type":"ContainerDied","Data":"b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059"}
Sep 09 21:55:35.441533 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.441506    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"
Sep 09 21:55:35.441859 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.441564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg" event={"ID":"5059685f-60a6-4168-bc7d-bec5ccf5ff4b","Type":"ContainerDied","Data":"9c59e76021b8e7bfdf4bc4252667d7dfb69fa44a0feb7f78785389eb7ef2aaa8"}
Sep 09 21:55:35.441859 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.441602    2563 scope.go:117] "RemoveContainer" containerID="b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059"
Sep 09 21:55:35.452235 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.452213    2563 scope.go:117] "RemoveContainer" containerID="b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059"
Sep 09 21:55:35.452513 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:55:35.452497    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059\": container with ID starting with b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059 not found: ID does not exist" containerID="b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059"
Sep 09 21:55:35.452566 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.452520    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059"} err="failed to get container status \"b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059\": rpc error: code = NotFound desc = could not find container \"b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059\": container with ID starting with b6090b6671d607493dca80b0a44fb70842c9429a218383239a787b2499dd6059 not found: ID does not exist"
Sep 09 21:55:35.466514 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.466486    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"]
Sep 09 21:55:35.469980 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.469958    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff5f88f5-5f986b8cff-tk8mg"]
Sep 09 21:55:35.606448 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.606418    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5059685f-60a6-4168-bc7d-bec5ccf5ff4b" path="/var/lib/kubelet/pods/5059685f-60a6-4168-bc7d-bec5ccf5ff4b/volumes"
Sep 09 21:55:35.625079 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.625044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:35.726439 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.726348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:35.828394 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:35.828361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:36.161688 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:36.161654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:36.259233 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:36.259199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:38.380859 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:38.380827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:38.486218 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:38.486178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:38.596419 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:38.596389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:39.046274 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.046236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:39.151923 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.151887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:39.221599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.221561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:39.265128 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.265095    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"]
Sep 09 21:55:39.265547 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.265535    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5059685f-60a6-4168-bc7d-bec5ccf5ff4b" containerName="maas-api"
Sep 09 21:55:39.265601 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.265549    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5059685f-60a6-4168-bc7d-bec5ccf5ff4b" containerName="maas-api"
Sep 09 21:55:39.265635 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.265621    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5059685f-60a6-4168-bc7d-bec5ccf5ff4b" containerName="maas-api"
Sep 09 21:55:39.268276 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.268261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:55:39.272289 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.272264    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9e97e70d\""
Sep 09 21:55:39.272289 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.272264    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9e97e70d-dockercfg-qh555\""
Sep 09 21:55:39.291181 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.291144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"]
Sep 09 21:55:39.438727 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.438646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstq8\" (UniqueName: \"kubernetes.io/projected/eb2f82e2-6197-4f44-9dc3-0db44cf25790-kube-api-access-gstq8\") pod \"payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2\" (UID: \"eb2f82e2-6197-4f44-9dc3-0db44cf25790\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:55:39.438727 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.438701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2f82e2-6197-4f44-9dc3-0db44cf25790-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2\" (UID: \"eb2f82e2-6197-4f44-9dc3-0db44cf25790\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:55:39.539495 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.539453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2f82e2-6197-4f44-9dc3-0db44cf25790-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2\" (UID: \"eb2f82e2-6197-4f44-9dc3-0db44cf25790\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:55:39.539667 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.539552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstq8\" (UniqueName: \"kubernetes.io/projected/eb2f82e2-6197-4f44-9dc3-0db44cf25790-kube-api-access-gstq8\") pod \"payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2\" (UID: \"eb2f82e2-6197-4f44-9dc3-0db44cf25790\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:55:39.540194 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.540173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2f82e2-6197-4f44-9dc3-0db44cf25790-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2\" (UID: \"eb2f82e2-6197-4f44-9dc3-0db44cf25790\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:55:39.548859 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.548830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstq8\" (UniqueName: \"kubernetes.io/projected/eb2f82e2-6197-4f44-9dc3-0db44cf25790-kube-api-access-gstq8\") pod \"payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2\" (UID: \"eb2f82e2-6197-4f44-9dc3-0db44cf25790\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:55:39.578548 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.578520    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:55:39.696429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.696340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:39.856042 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:39.856012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"]
Sep 09 21:55:39.857064 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:55:39.857038    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2f82e2_6197_4f44_9dc3_0db44cf25790.slice/crio-d7f7b84b303a2f49931cdbbfc14f7af7a003a377f08d9a49293ee1e2049a186d WatchSource:0}: Error finding container d7f7b84b303a2f49931cdbbfc14f7af7a003a377f08d9a49293ee1e2049a186d: Status 404 returned error can't find the container with id d7f7b84b303a2f49931cdbbfc14f7af7a003a377f08d9a49293ee1e2049a186d
Sep 09 21:55:40.476383 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:40.476342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2" event={"ID":"eb2f82e2-6197-4f44-9dc3-0db44cf25790","Type":"ContainerStarted","Data":"b5944a000165e48635ea8971e14c2aaeeadf8cb996e01bd5aa8192c73b6cd707"}
Sep 09 21:55:40.476824 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:40.476406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2" event={"ID":"eb2f82e2-6197-4f44-9dc3-0db44cf25790","Type":"ContainerStarted","Data":"d7f7b84b303a2f49931cdbbfc14f7af7a003a377f08d9a49293ee1e2049a186d"}
Sep 09 21:55:40.476824 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:40.476452    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:55:40.494961 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:40.494915    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2" podStartSLOduration=1.494897012 podStartE2EDuration="1.494897012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:40.493566878 +0000 UTC m=+1033.390232889" watchObservedRunningTime="2026-09-09 21:55:40.494897012 +0000 UTC m=+1033.391563021"
Sep 09 21:55:40.506413 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:40.506385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:41.171353 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:41.171312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:41.608108 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:41.608077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:51.484948 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:51.484916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:55:59.681599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.681538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:59.744149 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.744112    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"]
Sep 09 21:55:59.748053 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.748032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.750449 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.750426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-8c5761ec-kserve-self-signed-certs\""
Sep 09 21:55:59.758572 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.758550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"]
Sep 09 21:55:59.826372 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.826313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-home\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.826685 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.826618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tmp-dir\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.826918 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.826901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-model-cache\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.827001 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.826954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxjx\" (UniqueName: \"kubernetes.io/projected/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-kube-api-access-tkxjx\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.827058 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.827030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-dshm\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.827116 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.827096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tls-certs\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.884335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.884299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:59.928689 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.928651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxjx\" (UniqueName: \"kubernetes.io/projected/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-kube-api-access-tkxjx\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.928893 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.928724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-dshm\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.928893 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.928819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tls-certs\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.928893 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.928888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-home\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.929113 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.928914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tmp-dir\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.929113 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.928962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-model-cache\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.929376 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.929351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tmp-dir\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.929542 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.929390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-model-cache\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.929905 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.929445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-home\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.931671 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.931600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-dshm\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.932707 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.932668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tls-certs\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:55:59.937323 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:55:59.937301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxjx\" (UniqueName: \"kubernetes.io/projected/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-kube-api-access-tkxjx\") pod \"e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") " pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:56:00.057635 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:00.057599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:56:00.353278 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:00.353234    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"]
Sep 09 21:56:00.597073 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:00.597034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx" event={"ID":"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c","Type":"ContainerStarted","Data":"9c56e7a65f3f50d7ec62175ba7dc8da28cbe39591bdd1c56c944c78238dfc8cc"}
Sep 09 21:56:00.597241 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:00.597084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx" event={"ID":"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c","Type":"ContainerStarted","Data":"3dfa45f5b751f0bd178a6c660da543c45bb01ae8881892fe845a23f74625e853"}
Sep 09 21:56:00.614831 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:00.614681    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx" podStartSLOduration=1.614665017 podStartE2EDuration="1.614665017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:00.614075512 +0000 UTC m=+1053.510741521" watchObservedRunningTime="2026-09-09 21:56:00.614665017 +0000 UTC m=+1053.511331026"
Sep 09 21:56:09.357872 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.357746    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"]
Sep 09 21:56:09.361583 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.361559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.365602 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.365567    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-594ab9\"/\"openshift-service-ca.crt\""
Sep 09 21:56:09.365739 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.365683    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-594ab9\"/\"kube-root-ca.crt\""
Sep 09 21:56:09.365821 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.365748    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-594ab9\"/\"default-dockercfg-khjn2\""
Sep 09 21:56:09.366100 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.366012    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-594ab9\"/\"autores-model-594ab9-kserve-self-signed-certs\""
Sep 09 21:56:09.369757 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.369736    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"]
Sep 09 21:56:09.383366 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.383332    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:09.423119 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.423083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2v5x\" (UniqueName: \"kubernetes.io/projected/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-kube-api-access-z2v5x\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.423119 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.423135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-dshm\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.423356 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.423185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tls-certs\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.423356 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.423206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tmp-dir\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.423356 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.423296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-home\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.423356 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.423340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-model-cache\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.459712 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.459679    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"]
Sep 09 21:56:09.459983 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.459935    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2" podUID="eb2f82e2-6197-4f44-9dc3-0db44cf25790" containerName="payload-processing" containerID="cri-o://b5944a000165e48635ea8971e14c2aaeeadf8cb996e01bd5aa8192c73b6cd707" gracePeriod=30
Sep 09 21:56:09.461338 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.460878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:09.524426 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.524393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-dshm\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.524606 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.524447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tls-certs\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.524606 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.524473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tmp-dir\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.524606 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.524520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-home\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.524606 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.524573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-model-cache\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.524850 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.524643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2v5x\" (UniqueName: \"kubernetes.io/projected/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-kube-api-access-z2v5x\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.524940 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.524917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tmp-dir\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.525059 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.525037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-model-cache\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.525297 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.525278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-home\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.527077 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.527051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-dshm\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.527252 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.527235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tls-certs\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.531749 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.531722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:09.533902 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.533878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2v5x\" (UniqueName: \"kubernetes.io/projected/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-kube-api-access-z2v5x\") pod \"autores-model-594ab9-kserve-5b7487b7bc-vcghj\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") " pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.638692 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.638660    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:09.654379 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.654348    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb2f82e2-6197-4f44-9dc3-0db44cf25790" containerID="b5944a000165e48635ea8971e14c2aaeeadf8cb996e01bd5aa8192c73b6cd707" exitCode=0
Sep 09 21:56:09.654611 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.654584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2" event={"ID":"eb2f82e2-6197-4f44-9dc3-0db44cf25790","Type":"ContainerDied","Data":"b5944a000165e48635ea8971e14c2aaeeadf8cb996e01bd5aa8192c73b6cd707"}
Sep 09 21:56:09.673690 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.673655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:09.730188 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.729830    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:56:09.821255 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.821226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:09.828420 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.828401    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gstq8\" (UniqueName: \"kubernetes.io/projected/eb2f82e2-6197-4f44-9dc3-0db44cf25790-kube-api-access-gstq8\") pod \"eb2f82e2-6197-4f44-9dc3-0db44cf25790\" (UID: \"eb2f82e2-6197-4f44-9dc3-0db44cf25790\") "
Sep 09 21:56:09.828503 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.828470    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2f82e2-6197-4f44-9dc3-0db44cf25790-plugins-config-volume\") pod \"eb2f82e2-6197-4f44-9dc3-0db44cf25790\" (UID: \"eb2f82e2-6197-4f44-9dc3-0db44cf25790\") "
Sep 09 21:56:09.828884 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.828860    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2f82e2-6197-4f44-9dc3-0db44cf25790-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb2f82e2-6197-4f44-9dc3-0db44cf25790" (UID: "eb2f82e2-6197-4f44-9dc3-0db44cf25790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:09.830778 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.830748    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2f82e2-6197-4f44-9dc3-0db44cf25790-kube-api-access-gstq8" (OuterVolumeSpecName: "kube-api-access-gstq8") pod "eb2f82e2-6197-4f44-9dc3-0db44cf25790" (UID: "eb2f82e2-6197-4f44-9dc3-0db44cf25790"). InnerVolumeSpecName "kube-api-access-gstq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:09.918377 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.918289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:09.929634 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.929608    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstq8\" (UniqueName: \"kubernetes.io/projected/eb2f82e2-6197-4f44-9dc3-0db44cf25790-kube-api-access-gstq8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:09.929634 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.929633    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2f82e2-6197-4f44-9dc3-0db44cf25790-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:09.956456 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:09.956409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"]
Sep 09 21:56:09.958895 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:56:09.958866    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe5b3355_a71e_401d_b5f6_0d6e687b74fd.slice/crio-c14756a776035c2a2ea235226ca60fd2f4f0b13959ea6d2c80dcf2670174c9c9 WatchSource:0}: Error finding container c14756a776035c2a2ea235226ca60fd2f4f0b13959ea6d2c80dcf2670174c9c9: Status 404 returned error can't find the container with id c14756a776035c2a2ea235226ca60fd2f4f0b13959ea6d2c80dcf2670174c9c9
Sep 09 21:56:10.058722 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.058689    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:56:10.058929 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.058746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:56:10.071407 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.071377    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:56:10.288353 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.288318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:10.580753 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.580667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:10.664216 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.664174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj" event={"ID":"fe5b3355-a71e-401d-b5f6-0d6e687b74fd","Type":"ContainerStarted","Data":"a2d2896b21276e4e4dba82345b08fdbf051aeadf0daf54ad24d75e8ca1a00cd1"}
Sep 09 21:56:10.664420 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.664223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj" event={"ID":"fe5b3355-a71e-401d-b5f6-0d6e687b74fd","Type":"ContainerStarted","Data":"c14756a776035c2a2ea235226ca60fd2f4f0b13959ea6d2c80dcf2670174c9c9"}
Sep 09 21:56:10.665745 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.665722    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"
Sep 09 21:56:10.665899 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.665745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2" event={"ID":"eb2f82e2-6197-4f44-9dc3-0db44cf25790","Type":"ContainerDied","Data":"d7f7b84b303a2f49931cdbbfc14f7af7a003a377f08d9a49293ee1e2049a186d"}
Sep 09 21:56:10.665899 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.665808    2563 scope.go:117] "RemoveContainer" containerID="b5944a000165e48635ea8971e14c2aaeeadf8cb996e01bd5aa8192c73b6cd707"
Sep 09 21:56:10.680071 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.680045    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:56:10.684130 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.684090    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj" podStartSLOduration=1.684076707 podStartE2EDuration="1.684076707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:10.682327317 +0000 UTC m=+1063.578993352" watchObservedRunningTime="2026-09-09 21:56:10.684076707 +0000 UTC m=+1063.580742754"
Sep 09 21:56:10.724837 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.724798    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"]
Sep 09 21:56:10.731098 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.731071    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e97e70d-6f6999cbbb-wjzj2"]
Sep 09 21:56:10.898111 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:10.898030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:11.607213 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:11.607174    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb2f82e2-6197-4f44-9dc3-0db44cf25790" path="/var/lib/kubelet/pods/eb2f82e2-6197-4f44-9dc3-0db44cf25790/volumes"
Sep 09 21:56:13.127716 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:13.127613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:13.220341 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:13.220308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:14.497011 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:14.496981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:14.599246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:14.599213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:14.746466 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:14.746428    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:15.230101 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:15.229975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:15.319562 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:15.319526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:19.674395 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:19.674302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:19.674395 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:19.674354    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:19.686957 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:19.686930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:19.733583 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:19.733554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:21.032792 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.032735    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"]
Sep 09 21:56:21.033237 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.033223    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb2f82e2-6197-4f44-9dc3-0db44cf25790" containerName="payload-processing"
Sep 09 21:56:21.033282 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.033239    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb2f82e2-6197-4f44-9dc3-0db44cf25790" containerName="payload-processing"
Sep 09 21:56:21.033339 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.033328    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb2f82e2-6197-4f44-9dc3-0db44cf25790" containerName="payload-processing"
Sep 09 21:56:21.037866 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.037849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.040907 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.040882    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5aaa80\"/\"openshift-service-ca.crt\""
Sep 09 21:56:21.041710 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.041687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"]
Sep 09 21:56:21.042178 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.042155    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5aaa80\"/\"kube-root-ca.crt\""
Sep 09 21:56:21.042289 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.042188    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5aaa80\"/\"default-dockercfg-n62gn\""
Sep 09 21:56:21.042289 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.042209    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5aaa80\"/\"test-model-5aaa80-kserve-self-signed-certs\""
Sep 09 21:56:21.146287 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.146245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-dshm\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.146478 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.146303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-home\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.146478 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.146352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4hfw\" (UniqueName: \"kubernetes.io/projected/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-kube-api-access-m4hfw\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.146478 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.146448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tmp-dir\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.146584 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.146488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-model-cache\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.146584 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.146527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tls-certs\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.247633 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.247594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-home\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.247633 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.247633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4hfw\" (UniqueName: \"kubernetes.io/projected/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-kube-api-access-m4hfw\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.247898 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.247664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tmp-dir\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.247898 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.247705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-model-cache\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.247898 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.247759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tls-certs\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.247898 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.247872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-dshm\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.248116 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.248063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-home\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.248116 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.248097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tmp-dir\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.248228 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.248138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-model-cache\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.250234 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.250215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-dshm\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.250614 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.250595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tls-certs\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.257127 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.257108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4hfw\" (UniqueName: \"kubernetes.io/projected/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-kube-api-access-m4hfw\") pod \"test-model-5aaa80-kserve-7d46b9dccc-wplm8\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") " pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.349257 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.349155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:21.443831 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.443796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:21.624021 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.623998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"]
Sep 09 21:56:21.625674 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:56:21.625644    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8e3ae77_b83d_4d36_843f_7b6a36445ec2.slice/crio-a9cbdce282d3f196778583418a7769007c509dbaba1dbbbe034b9847dd7ddf22 WatchSource:0}: Error finding container a9cbdce282d3f196778583418a7769007c509dbaba1dbbbe034b9847dd7ddf22: Status 404 returned error can't find the container with id a9cbdce282d3f196778583418a7769007c509dbaba1dbbbe034b9847dd7ddf22
Sep 09 21:56:21.738090 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.738051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8" event={"ID":"d8e3ae77-b83d-4d36-843f-7b6a36445ec2","Type":"ContainerStarted","Data":"f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e"}
Sep 09 21:56:21.738246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.738099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8" event={"ID":"d8e3ae77-b83d-4d36-843f-7b6a36445ec2","Type":"ContainerStarted","Data":"a9cbdce282d3f196778583418a7769007c509dbaba1dbbbe034b9847dd7ddf22"}
Sep 09 21:56:21.755686 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:21.755637    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8" podStartSLOduration=0.755620627 podStartE2EDuration="755.620627ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:21.754837186 +0000 UTC m=+1074.651503195" watchObservedRunningTime="2026-09-09 21:56:21.755620627 +0000 UTC m=+1074.652286636"
Sep 09 21:56:27.662297 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:27.662243    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"]
Sep 09 21:56:27.699514 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:27.699476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:31.349452 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:31.349407    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:31.349893 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:31.349475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:31.362450 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:31.362422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:31.731881 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:31.731786    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"]
Sep 09 21:56:31.754758 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:31.754727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:31.807729 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:31.807701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:56:31.995837 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:31.995806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:36.848296 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:36.848263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:42.004647 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.004608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:42.620401 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.620368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:42.659027 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.658992    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"]
Sep 09 21:56:42.659221 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.659194    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v" podUID="255ed03c-02f4-44cc-9cf4-4d7f2eaed76c" containerName="payload-pre-processing" containerID="cri-o://42bc95916c699bc744b539d2f28282e443a537487bdd05c3be2f3ba1bc615309" gracePeriod=30
Sep 09 21:56:42.692232 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.692202    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx" podUID="ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" containerName="main" containerID="cri-o://9c56e7a65f3f50d7ec62175ba7dc8da28cbe39591bdd1c56c944c78238dfc8cc" gracePeriod=45
Sep 09 21:56:42.721540 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.721496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:42.824403 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.824372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:42.867285 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.867170    2563 generic.go:356] "Generic (PLEG): container finished" podID="255ed03c-02f4-44cc-9cf4-4d7f2eaed76c" containerID="42bc95916c699bc744b539d2f28282e443a537487bdd05c3be2f3ba1bc615309" exitCode=0
Sep 09 21:56:42.867285 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.867250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v" event={"ID":"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c","Type":"ContainerDied","Data":"42bc95916c699bc744b539d2f28282e443a537487bdd05c3be2f3ba1bc615309"}
Sep 09 21:56:42.870302 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.870169    2563 generic.go:356] "Generic (PLEG): container finished" podID="ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" containerID="9c56e7a65f3f50d7ec62175ba7dc8da28cbe39591bdd1c56c944c78238dfc8cc" exitCode=0
Sep 09 21:56:42.870556 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.870475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx" event={"ID":"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c","Type":"ContainerDied","Data":"9c56e7a65f3f50d7ec62175ba7dc8da28cbe39591bdd1c56c944c78238dfc8cc"}
Sep 09 21:56:42.923817 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.923789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:42.950624 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.950599    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:56:42.977359 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:42.977331    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:56:43.072999 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.072966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkxjx\" (UniqueName: \"kubernetes.io/projected/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-kube-api-access-tkxjx\") pod \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") "
Sep 09 21:56:43.073477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073008    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-home\") pod \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") "
Sep 09 21:56:43.073477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073036    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-plugins-config-volume\") pod \"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c\" (UID: \"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c\") "
Sep 09 21:56:43.073477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tmp-dir\") pod \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") "
Sep 09 21:56:43.073477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd5sj\" (UniqueName: \"kubernetes.io/projected/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-kube-api-access-cd5sj\") pod \"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c\" (UID: \"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c\") "
Sep 09 21:56:43.073477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073157    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-dshm\") pod \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") "
Sep 09 21:56:43.073477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073174    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tls-certs\") pod \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") "
Sep 09 21:56:43.073477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-model-cache\") pod \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\" (UID: \"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c\") "
Sep 09 21:56:43.073477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-home" (OuterVolumeSpecName: "home") pod "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" (UID: "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:43.073477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073327    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" (UID: "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:43.073883 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073529    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" (UID: "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:43.073883 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073582    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:43.073883 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073636    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:43.073883 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.073654    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "255ed03c-02f4-44cc-9cf4-4d7f2eaed76c" (UID: "255ed03c-02f4-44cc-9cf4-4d7f2eaed76c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:43.075528 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.075507    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-dshm" (OuterVolumeSpecName: "dshm") pod "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" (UID: "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:43.075611 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.075584    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" (UID: "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:56:43.075954 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.075936    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-kube-api-access-tkxjx" (OuterVolumeSpecName: "kube-api-access-tkxjx") pod "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" (UID: "ecafa27d-1dc7-4594-b2c1-fc4e55ea533c"). InnerVolumeSpecName "kube-api-access-tkxjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:43.076003 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.075964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-kube-api-access-cd5sj" (OuterVolumeSpecName: "kube-api-access-cd5sj") pod "255ed03c-02f4-44cc-9cf4-4d7f2eaed76c" (UID: "255ed03c-02f4-44cc-9cf4-4d7f2eaed76c"). InnerVolumeSpecName "kube-api-access-cd5sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:43.175051 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.174957    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:43.175051 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.174986    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:43.175051 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.175002    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:43.175051 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.175013    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxjx\" (UniqueName: \"kubernetes.io/projected/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c-kube-api-access-tkxjx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:43.175051 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.175028    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:43.175051 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.175041    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd5sj\" (UniqueName: \"kubernetes.io/projected/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c-kube-api-access-cd5sj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:43.227693 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.227662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:43.878220 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.878121    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"
Sep 09 21:56:43.878220 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.878129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v" event={"ID":"255ed03c-02f4-44cc-9cf4-4d7f2eaed76c","Type":"ContainerDied","Data":"1a108ab06df245dbff3fc084a3adae2f4332b604b26e950679d28315ed7e00e9"}
Sep 09 21:56:43.878220 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.878193    2563 scope.go:117] "RemoveContainer" containerID="42bc95916c699bc744b539d2f28282e443a537487bdd05c3be2f3ba1bc615309"
Sep 09 21:56:43.879634 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.879623    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"
Sep 09 21:56:43.879722 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.879619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx" event={"ID":"ecafa27d-1dc7-4594-b2c1-fc4e55ea533c","Type":"ContainerDied","Data":"3dfa45f5b751f0bd178a6c660da543c45bb01ae8881892fe845a23f74625e853"}
Sep 09 21:56:43.887878 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.887863    2563 scope.go:117] "RemoveContainer" containerID="9c56e7a65f3f50d7ec62175ba7dc8da28cbe39591bdd1c56c944c78238dfc8cc"
Sep 09 21:56:43.901972 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.901945    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"]
Sep 09 21:56:43.904653 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.904631    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-8c5761ec-kserve-69b45c47dc-5zlnx"]
Sep 09 21:56:43.915589 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.915564    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"]
Sep 09 21:56:43.921927 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:43.921901    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-594ab9-8656d67cc4-j4h2v"]
Sep 09 21:56:44.598348 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:44.598310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:45.466624 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:45.466572    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:45.607289 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:45.607245    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="255ed03c-02f4-44cc-9cf4-4d7f2eaed76c" path="/var/lib/kubelet/pods/255ed03c-02f4-44cc-9cf4-4d7f2eaed76c/volumes"
Sep 09 21:56:45.607849 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:45.607830    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" path="/var/lib/kubelet/pods/ecafa27d-1dc7-4594-b2c1-fc4e55ea533c/volumes"
Sep 09 21:56:45.631745 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:45.630429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:45.768029 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:45.767987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:45.884569 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:45.884529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:45.986229 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:45.986188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:46.752671 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:46.752638    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj" podUID="fe5b3355-a71e-401d-b5f6-0d6e687b74fd" containerName="main" containerID="cri-o://a2d2896b21276e4e4dba82345b08fdbf051aeadf0daf54ad24d75e8ca1a00cd1" gracePeriod=45
Sep 09 21:56:46.902993 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:46.902961    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe5b3355-a71e-401d-b5f6-0d6e687b74fd" containerID="a2d2896b21276e4e4dba82345b08fdbf051aeadf0daf54ad24d75e8ca1a00cd1" exitCode=0
Sep 09 21:56:46.903201 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:46.903006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj" event={"ID":"fe5b3355-a71e-401d-b5f6-0d6e687b74fd","Type":"ContainerDied","Data":"a2d2896b21276e4e4dba82345b08fdbf051aeadf0daf54ad24d75e8ca1a00cd1"}
Sep 09 21:56:47.026294 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.026269    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:47.068991 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.068949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:47.112303 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112267    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-dshm\") pod \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") "
Sep 09 21:56:47.112494 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112340    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tmp-dir\") pod \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") "
Sep 09 21:56:47.112494 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112368    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2v5x\" (UniqueName: \"kubernetes.io/projected/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-kube-api-access-z2v5x\") pod \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") "
Sep 09 21:56:47.112494 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112389    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-home\") pod \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") "
Sep 09 21:56:47.112494 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tls-certs\") pod \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") "
Sep 09 21:56:47.112494 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-model-cache\") pod \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\" (UID: \"fe5b3355-a71e-401d-b5f6-0d6e687b74fd\") "
Sep 09 21:56:47.112764 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112651    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe5b3355-a71e-401d-b5f6-0d6e687b74fd" (UID: "fe5b3355-a71e-401d-b5f6-0d6e687b74fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:47.112764 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112673    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-home" (OuterVolumeSpecName: "home") pod "fe5b3355-a71e-401d-b5f6-0d6e687b74fd" (UID: "fe5b3355-a71e-401d-b5f6-0d6e687b74fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:47.112925 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112798    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe5b3355-a71e-401d-b5f6-0d6e687b74fd" (UID: "fe5b3355-a71e-401d-b5f6-0d6e687b74fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:47.113138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112966    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:47.113138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.112988    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:47.113138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.113001    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:47.115400 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.115360    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-kube-api-access-z2v5x" (OuterVolumeSpecName: "kube-api-access-z2v5x") pod "fe5b3355-a71e-401d-b5f6-0d6e687b74fd" (UID: "fe5b3355-a71e-401d-b5f6-0d6e687b74fd"). InnerVolumeSpecName "kube-api-access-z2v5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:47.115540 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.115401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe5b3355-a71e-401d-b5f6-0d6e687b74fd" (UID: "fe5b3355-a71e-401d-b5f6-0d6e687b74fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:56:47.115891 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.115865    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-dshm" (OuterVolumeSpecName: "dshm") pod "fe5b3355-a71e-401d-b5f6-0d6e687b74fd" (UID: "fe5b3355-a71e-401d-b5f6-0d6e687b74fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:47.213687 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.213655    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:47.213687 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.213682    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2v5x\" (UniqueName: \"kubernetes.io/projected/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-kube-api-access-z2v5x\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:47.213687 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.213693    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5b3355-a71e-401d-b5f6-0d6e687b74fd-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:47.472000 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.471917    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:47.786429 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.786384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:47.910568 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.910543    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"
Sep 09 21:56:47.910746 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.910572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj" event={"ID":"fe5b3355-a71e-401d-b5f6-0d6e687b74fd","Type":"ContainerDied","Data":"c14756a776035c2a2ea235226ca60fd2f4f0b13959ea6d2c80dcf2670174c9c9"}
Sep 09 21:56:47.910746 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.910619    2563 scope.go:117] "RemoveContainer" containerID="a2d2896b21276e4e4dba82345b08fdbf051aeadf0daf54ad24d75e8ca1a00cd1"
Sep 09 21:56:47.935915 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.935882    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"]
Sep 09 21:56:47.941936 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.941909    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-594ab9/autores-model-594ab9-kserve-5b7487b7bc-vcghj"]
Sep 09 21:56:47.987388 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:47.987361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:49.607172 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:49.607139    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5b3355-a71e-401d-b5f6-0d6e687b74fd" path="/var/lib/kubelet/pods/fe5b3355-a71e-401d-b5f6-0d6e687b74fd/volumes"
Sep 09 21:56:49.953262 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:49.953186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:50.059742 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:50.059712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:50.211455 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:50.211378    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"]
Sep 09 21:56:50.211663 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:50.211629    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy" containerID="cri-o://83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229" gracePeriod=30
Sep 09 21:56:50.263269 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:50.263213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:50.689199 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:50.689167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:50.800576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:50.800533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:51.181297 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:51.181262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:51.433138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:51.433041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:51.973529 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:51.973497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:52.087602 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:52.087569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:52.417355 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:52.417327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:52.795162 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:52.795126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:52.931419 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:52.931384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:53.366617 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:53.366564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:54.403416 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:54.403384    2563 patch_prober.go:28] interesting pod/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 21:56:54.403810 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:54.403441    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:56:54.527134 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:54.527103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:54.623156 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:54.623125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:55.470057 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.470034    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:56:55.591339 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591306    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-certs\") pod \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") "
Sep 09 21:56:55.591548 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-token\") pod \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") "
Sep 09 21:56:55.591548 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591411    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-data\") pod \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") "
Sep 09 21:56:55.591548 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591436    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-podinfo\") pod \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") "
Sep 09 21:56:55.591548 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-credential-socket\") pod \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") "
Sep 09 21:56:55.591548 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591512    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h45w6\" (UniqueName: \"kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-kube-api-access-h45w6\") pod \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") "
Sep 09 21:56:55.591850 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-envoy\") pod \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") "
Sep 09 21:56:55.591850 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591621    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-socket\") pod \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") "
Sep 09 21:56:55.591850 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591651    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04c59c6d-a4ef-4bd5-ba64-f2df6d176351" (UID: "04c59c6d-a4ef-4bd5-ba64-f2df6d176351"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:55.591850 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591682    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istiod-ca-cert\") pod \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\" (UID: \"04c59c6d-a4ef-4bd5-ba64-f2df6d176351\") "
Sep 09 21:56:55.591850 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591733    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-data" (OuterVolumeSpecName: "istio-data") pod "04c59c6d-a4ef-4bd5-ba64-f2df6d176351" (UID: "04c59c6d-a4ef-4bd5-ba64-f2df6d176351"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:55.591850 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.591805    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04c59c6d-a4ef-4bd5-ba64-f2df6d176351" (UID: "04c59c6d-a4ef-4bd5-ba64-f2df6d176351"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:55.592132 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.592065    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04c59c6d-a4ef-4bd5-ba64-f2df6d176351" (UID: "04c59c6d-a4ef-4bd5-ba64-f2df6d176351"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:55.592132 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.592101    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:55.592132 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.592123    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:55.592266 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.592136    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:55.592266 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.592176    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04c59c6d-a4ef-4bd5-ba64-f2df6d176351" (UID: "04c59c6d-a4ef-4bd5-ba64-f2df6d176351"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:55.594080 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.594046    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04c59c6d-a4ef-4bd5-ba64-f2df6d176351" (UID: "04c59c6d-a4ef-4bd5-ba64-f2df6d176351"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:55.594301 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.594269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-kube-api-access-h45w6" (OuterVolumeSpecName: "kube-api-access-h45w6") pod "04c59c6d-a4ef-4bd5-ba64-f2df6d176351" (UID: "04c59c6d-a4ef-4bd5-ba64-f2df6d176351"). InnerVolumeSpecName "kube-api-access-h45w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:55.594411 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.594337    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-token" (OuterVolumeSpecName: "istio-token") pod "04c59c6d-a4ef-4bd5-ba64-f2df6d176351" (UID: "04c59c6d-a4ef-4bd5-ba64-f2df6d176351"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:55.594411 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.594362    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04c59c6d-a4ef-4bd5-ba64-f2df6d176351" (UID: "04c59c6d-a4ef-4bd5-ba64-f2df6d176351"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:56:55.693635 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.693592    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:55.693635 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.693628    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:55.693635 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.693637    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:55.693635 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.693646    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:55.693944 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.693654    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h45w6\" (UniqueName: \"kubernetes.io/projected/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-kube-api-access-h45w6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:55.693944 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.693663    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c59c6d-a4ef-4bd5-ba64-f2df6d176351-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:56:55.774830 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.774797    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"]
Sep 09 21:56:55.775340 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775315    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy"
Sep 09 21:56:55.775340 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775342    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy"
Sep 09 21:56:55.775597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775381    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" containerName="main"
Sep 09 21:56:55.775597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775389    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" containerName="main"
Sep 09 21:56:55.775597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775399    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5b3355-a71e-401d-b5f6-0d6e687b74fd" containerName="main"
Sep 09 21:56:55.775597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775407    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5b3355-a71e-401d-b5f6-0d6e687b74fd" containerName="main"
Sep 09 21:56:55.775597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775419    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy"
Sep 09 21:56:55.775597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775428    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy"
Sep 09 21:56:55.775597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775442    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="255ed03c-02f4-44cc-9cf4-4d7f2eaed76c" containerName="payload-pre-processing"
Sep 09 21:56:55.775597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775450    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="255ed03c-02f4-44cc-9cf4-4d7f2eaed76c" containerName="payload-pre-processing"
Sep 09 21:56:55.775597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775585    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecafa27d-1dc7-4594-b2c1-fc4e55ea533c" containerName="main"
Sep 09 21:56:55.776042 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775604    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy"
Sep 09 21:56:55.776042 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775613    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="255ed03c-02f4-44cc-9cf4-4d7f2eaed76c" containerName="payload-pre-processing"
Sep 09 21:56:55.776042 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775624    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5b3355-a71e-401d-b5f6-0d6e687b74fd" containerName="main"
Sep 09 21:56:55.776042 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.775633    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerName="istio-proxy"
Sep 09 21:56:55.779307 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.779289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.782553 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.782527    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-bbdd85-serving-cert\""
Sep 09 21:56:55.793553 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.793527    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"]
Sep 09 21:56:55.810841 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.810807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:55.884124 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.884033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:55.895252 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.895220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-metrics-certs\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.895374 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.895258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-maas-api-tls\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.895374 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.895283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vljkv\" (UniqueName: \"kubernetes.io/projected/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-kube-api-access-vljkv\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.895374 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.895357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-openshift-service-ca\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.962849 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.962816    2563 generic.go:356] "Generic (PLEG): container finished" podID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" containerID="83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229" exitCode=0
Sep 09 21:56:55.963030 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.962866    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"
Sep 09 21:56:55.963030 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.962897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" event={"ID":"04c59c6d-a4ef-4bd5-ba64-f2df6d176351","Type":"ContainerDied","Data":"83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229"}
Sep 09 21:56:55.963030 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.962950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl" event={"ID":"04c59c6d-a4ef-4bd5-ba64-f2df6d176351","Type":"ContainerDied","Data":"270a82946d83c5dfc1fdce74716f1b328a7579b37f4e245cac24d2647bdff4b2"}
Sep 09 21:56:55.963030 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.962973    2563 scope.go:117] "RemoveContainer" containerID="83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229"
Sep 09 21:56:55.972342 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.972326    2563 scope.go:117] "RemoveContainer" containerID="b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd"
Sep 09 21:56:55.980981 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.980959    2563 scope.go:117] "RemoveContainer" containerID="83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229"
Sep 09 21:56:55.981410 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:56:55.981388    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229\": container with ID starting with 83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229 not found: ID does not exist" containerID="83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229"
Sep 09 21:56:55.981521 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.981421    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229"} err="failed to get container status \"83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229\": rpc error: code = NotFound desc = could not find container \"83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229\": container with ID starting with 83805be67010cb742ee4288066efe13142aa92da23bdce1b531a2e1010b42229 not found: ID does not exist"
Sep 09 21:56:55.981521 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.981445    2563 scope.go:117] "RemoveContainer" containerID="b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd"
Sep 09 21:56:55.981820 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:56:55.981794    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd\": container with ID starting with b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd not found: ID does not exist" containerID="b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd"
Sep 09 21:56:55.981883 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.981828    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd"} err="failed to get container status \"b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd\": rpc error: code = NotFound desc = could not find container \"b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd\": container with ID starting with b86874576665bf6ea769f50e2d9e121a67af4420402a78692d19c9cd97bd35fd not found: ID does not exist"
Sep 09 21:56:55.984524 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.984507    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"]
Sep 09 21:56:55.984684 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.984652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:55.986989 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.986971    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-594ab9-openshift-default-5b8fb54758-jhjfl"]
Sep 09 21:56:55.995956 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.995934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-metrics-certs\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.996048 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.995969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-maas-api-tls\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.996048 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.995997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vljkv\" (UniqueName: \"kubernetes.io/projected/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-kube-api-access-vljkv\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.996048 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.996029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-openshift-service-ca\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.996728 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.996708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-openshift-service-ca\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.998425 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.998404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-metrics-certs\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:55.998492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:55.998469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-maas-api-tls\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:56.007063 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:56.007042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vljkv\" (UniqueName: \"kubernetes.io/projected/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-kube-api-access-vljkv\") pod \"maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:56.091832 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:56.091797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:56.323709 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:56.323672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:56.368171 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:56.368146    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"]
Sep 09 21:56:56.369491 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:56:56.369456    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a617c8_b3fa_4a80_85b3_a8b41d82b45a.slice/crio-2df4fd65a21fc9153f31f7e8cbd35c300204189a216f42769ab2ce7255b6d7a4 WatchSource:0}: Error finding container 2df4fd65a21fc9153f31f7e8cbd35c300204189a216f42769ab2ce7255b6d7a4: Status 404 returned error can't find the container with id 2df4fd65a21fc9153f31f7e8cbd35c300204189a216f42769ab2ce7255b6d7a4
Sep 09 21:56:56.433268 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:56.433179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:56.972349 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:56.972307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc" event={"ID":"99a617c8-b3fa-4a80-85b3-a8b41d82b45a","Type":"ContainerStarted","Data":"03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9"}
Sep 09 21:56:56.972841 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:56.972415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc" event={"ID":"99a617c8-b3fa-4a80-85b3-a8b41d82b45a","Type":"ContainerStarted","Data":"2df4fd65a21fc9153f31f7e8cbd35c300204189a216f42769ab2ce7255b6d7a4"}
Sep 09 21:56:56.972841 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:56.972449    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:56:56.992972 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:56.992871    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc" podStartSLOduration=1.6327322560000002 podStartE2EDuration="1.992855492s" totalImagesPullingTime="360.123236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:56.990599523 +0000 UTC m=+1109.887265552" watchObservedRunningTime="2026-09-09 21:56:56.992855492 +0000 UTC m=+1109.889521548"
Sep 09 21:56:57.606745 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:57.606708    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04c59c6d-a4ef-4bd5-ba64-f2df6d176351" path="/var/lib/kubelet/pods/04c59c6d-a4ef-4bd5-ba64-f2df6d176351/volumes"
Sep 09 21:56:59.827170 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:56:59.827137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:02.984517 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:02.984486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:57:06.661197 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:06.661159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:14.878943 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:14.878907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:16.910477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:16.910443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:19.367016 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:19.366975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:20.794760 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:20.794727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:23.887054 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.887022    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"]
Sep 09 21:57:23.892296 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.892276    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:23.895489 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.895462    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ebe9b0\"/\"kube-root-ca.crt\""
Sep 09 21:57:23.896474 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.896451    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ebe9b0\"/\"test-model-ebe9b0-kserve-self-signed-certs\""
Sep 09 21:57:23.896579 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.896506    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ebe9b0\"/\"openshift-service-ca.crt\""
Sep 09 21:57:23.896719 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.896706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ebe9b0\"/\"default-dockercfg-grzds\""
Sep 09 21:57:23.903245 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.903224    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"]
Sep 09 21:57:23.932398 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.932365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:23.945617 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.945580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-home\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:23.945732 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.945642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-dshm\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:23.945732 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.945664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26wq\" (UniqueName: \"kubernetes.io/projected/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-kube-api-access-s26wq\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:23.945732 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.945687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tls-certs\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:23.945732 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.945724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tmp-dir\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:23.945923 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:23.945805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-model-cache\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.046443 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.046403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-home\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.046654 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.046467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-dshm\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.046654 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.046501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26wq\" (UniqueName: \"kubernetes.io/projected/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-kube-api-access-s26wq\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.046654 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.046525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tls-certs\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.046654 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.046583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tmp-dir\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.046654 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.046626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-model-cache\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.047001 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.046955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-home\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.047058 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.047023    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tmp-dir\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.047111 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.047054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-model-cache\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.048995 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.048967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-dshm\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.049158 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.049142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tls-certs\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.059402 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.059377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26wq\" (UniqueName: \"kubernetes.io/projected/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-kube-api-access-s26wq\") pod \"test-model-ebe9b0-kserve-69d484598c-5bgkv\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") " pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.202445 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.202358    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:24.477614 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:24.477581    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"]
Sep 09 21:57:24.479208 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:57:24.479183    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe79d8aa_80e1_4bcd_9ecc_ee120d7560ad.slice/crio-0361783c596b1d27488638464ccb4f5edc785711451966f7a80eb66b87203d50 WatchSource:0}: Error finding container 0361783c596b1d27488638464ccb4f5edc785711451966f7a80eb66b87203d50: Status 404 returned error can't find the container with id 0361783c596b1d27488638464ccb4f5edc785711451966f7a80eb66b87203d50
Sep 09 21:57:25.138078 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:25.138043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv" event={"ID":"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad","Type":"ContainerStarted","Data":"3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9"}
Sep 09 21:57:25.138485 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:25.138087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv" event={"ID":"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad","Type":"ContainerStarted","Data":"0361783c596b1d27488638464ccb4f5edc785711451966f7a80eb66b87203d50"}
Sep 09 21:57:25.169813 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:25.169744    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv" podStartSLOduration=2.169727625 podStartE2EDuration="2.169727625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:57:25.165468684 +0000 UTC m=+1138.062134703" watchObservedRunningTime="2026-09-09 21:57:25.169727625 +0000 UTC m=+1138.066393633"
Sep 09 21:57:29.662170 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.662126    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"]
Sep 09 21:57:29.667463 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.667425    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.671528 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.671501    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bbdd85\"/\"explicit-model-bbdd85-kserve-self-signed-certs\""
Sep 09 21:57:29.671695 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.671555    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bbdd85\"/\"openshift-service-ca.crt\""
Sep 09 21:57:29.671695 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.671609    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bbdd85\"/\"default-dockercfg-22njg\""
Sep 09 21:57:29.671695 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.671562    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bbdd85\"/\"kube-root-ca.crt\""
Sep 09 21:57:29.675883 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.675655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"]
Sep 09 21:57:29.696214 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.696175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:29.807136 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.807100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-dshm\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.807348 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.807153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-home\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.807348 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.807174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tls-certs\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.807348 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.807250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw6hk\" (UniqueName: \"kubernetes.io/projected/9f6a9172-5491-4a00-aea2-11d0be8a6e46-kube-api-access-vw6hk\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.807495 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.807354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tmp-dir\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.807495 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.807391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-model-cache\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.908860 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.908822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tmp-dir\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.908860 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.908869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-model-cache\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.909168 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.908909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-dshm\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.909168 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.908956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-home\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.909168 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.908978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tls-certs\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.909168 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.909009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw6hk\" (UniqueName: \"kubernetes.io/projected/9f6a9172-5491-4a00-aea2-11d0be8a6e46-kube-api-access-vw6hk\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.909388 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.909372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-model-cache\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.909453 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.909392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-home\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.909453 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.909435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tmp-dir\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.911499 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.911453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-dshm\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.911946 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.911918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tls-certs\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.918229 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.918166    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw6hk\" (UniqueName: \"kubernetes.io/projected/9f6a9172-5491-4a00-aea2-11d0be8a6e46-kube-api-access-vw6hk\") pod \"explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") " pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:29.978603 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:29.978571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:30.266205 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:30.266168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"]
Sep 09 21:57:30.267756 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:57:30.267718    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f6a9172_5491_4a00_aea2_11d0be8a6e46.slice/crio-041ae247a7c976d0c567dc0109a15768f45279fb9df361fd3c281f104573e760 WatchSource:0}: Error finding container 041ae247a7c976d0c567dc0109a15768f45279fb9df361fd3c281f104573e760: Status 404 returned error can't find the container with id 041ae247a7c976d0c567dc0109a15768f45279fb9df361fd3c281f104573e760
Sep 09 21:57:31.178912 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:31.178875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8" event={"ID":"9f6a9172-5491-4a00-aea2-11d0be8a6e46","Type":"ContainerStarted","Data":"e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98"}
Sep 09 21:57:31.179375 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:31.178920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8" event={"ID":"9f6a9172-5491-4a00-aea2-11d0be8a6e46","Type":"ContainerStarted","Data":"041ae247a7c976d0c567dc0109a15768f45279fb9df361fd3c281f104573e760"}
Sep 09 21:57:31.197741 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:31.197692    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8" podStartSLOduration=2.197676915 podStartE2EDuration="2.197676915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:57:31.195741914 +0000 UTC m=+1144.092407922" watchObservedRunningTime="2026-09-09 21:57:31.197676915 +0000 UTC m=+1144.094342923"
Sep 09 21:57:32.766480 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:32.766448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:32.833874 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:32.833828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:32.902565 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:32.902509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:32.982185 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:32.982143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:33.121843 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:33.121723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:33.814350 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:33.814319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:34.003333 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:34.003302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:34.203322 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:34.203242    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:34.203322 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:34.203288    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:34.217580 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:34.217551    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:35.212508 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:35.212476    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:57:35.331389 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:35.331358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:39.978933 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:39.978895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:39.979400 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:39.978961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:39.991553 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:39.991519    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:40.244280 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:40.244252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:57:42.838095 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:42.838065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:42.947985 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:42.947952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:43.053332 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:43.053298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:43.152230 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:43.152151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:43.531025 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:43.530994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:44.941264 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:44.941227    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"]
Sep 09 21:57:44.973492 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:44.973458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:45.250039 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:45.250008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:55.559414 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.559323    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"]
Sep 09 21:57:55.559923 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.559594    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc" podUID="99a617c8-b3fa-4a80-85b3-a8b41d82b45a" containerName="maas-api" containerID="cri-o://03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9" gracePeriod=30
Sep 09 21:57:55.594490 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.594446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:55.817332 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.817264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:57:55.889165 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.889130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-maas-api-tls\") pod \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") "
Sep 09 21:57:55.889339 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.889176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vljkv\" (UniqueName: \"kubernetes.io/projected/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-kube-api-access-vljkv\") pod \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") "
Sep 09 21:57:55.889339 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.889238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-openshift-service-ca\") pod \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") "
Sep 09 21:57:55.889339 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.889278    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-metrics-certs\") pod \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\" (UID: \"99a617c8-b3fa-4a80-85b3-a8b41d82b45a\") "
Sep 09 21:57:55.889587 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.889560    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99a617c8-b3fa-4a80-85b3-a8b41d82b45a" (UID: "99a617c8-b3fa-4a80-85b3-a8b41d82b45a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:57:55.891576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.891546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "99a617c8-b3fa-4a80-85b3-a8b41d82b45a" (UID: "99a617c8-b3fa-4a80-85b3-a8b41d82b45a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:57:55.891576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.891560    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99a617c8-b3fa-4a80-85b3-a8b41d82b45a" (UID: "99a617c8-b3fa-4a80-85b3-a8b41d82b45a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:57:55.891711 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.891573    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-kube-api-access-vljkv" (OuterVolumeSpecName: "kube-api-access-vljkv") pod "99a617c8-b3fa-4a80-85b3-a8b41d82b45a" (UID: "99a617c8-b3fa-4a80-85b3-a8b41d82b45a"). InnerVolumeSpecName "kube-api-access-vljkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:57:55.990729 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.990690    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:57:55.990729 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.990723    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:57:55.990729 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.990733    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:57:55.990729 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:55.990740    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vljkv\" (UniqueName: \"kubernetes.io/projected/99a617c8-b3fa-4a80-85b3-a8b41d82b45a-kube-api-access-vljkv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:57:56.334145 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:56.334109    2563 generic.go:356] "Generic (PLEG): container finished" podID="99a617c8-b3fa-4a80-85b3-a8b41d82b45a" containerID="03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9" exitCode=0
Sep 09 21:57:56.334326 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:56.334155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc" event={"ID":"99a617c8-b3fa-4a80-85b3-a8b41d82b45a","Type":"ContainerDied","Data":"03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9"}
Sep 09 21:57:56.334326 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:56.334160    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"
Sep 09 21:57:56.334326 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:56.334198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc" event={"ID":"99a617c8-b3fa-4a80-85b3-a8b41d82b45a","Type":"ContainerDied","Data":"2df4fd65a21fc9153f31f7e8cbd35c300204189a216f42769ab2ce7255b6d7a4"}
Sep 09 21:57:56.334326 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:56.334221    2563 scope.go:117] "RemoveContainer" containerID="03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9"
Sep 09 21:57:56.345342 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:56.345312    2563 scope.go:117] "RemoveContainer" containerID="03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9"
Sep 09 21:57:56.345657 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:57:56.345636    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9\": container with ID starting with 03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9 not found: ID does not exist" containerID="03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9"
Sep 09 21:57:56.345749 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:56.345664    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9"} err="failed to get container status \"03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9\": rpc error: code = NotFound desc = could not find container \"03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9\": container with ID starting with 03dc1aefb5f161e69851e23f669cda7cdd0a5b40746a8b97f1271d9ecfe48cb9 not found: ID does not exist"
Sep 09 21:57:56.368025 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:56.367997    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"]
Sep 09 21:57:56.374265 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:56.374230    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bbdd85-6d787cb454-5xsfc"]
Sep 09 21:57:56.979503 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:56.979470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:57.606445 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:57.606416    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99a617c8-b3fa-4a80-85b3-a8b41d82b45a" path="/var/lib/kubelet/pods/99a617c8-b3fa-4a80-85b3-a8b41d82b45a/volumes"
Sep 09 21:57:59.961420 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:57:59.961391    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8" podUID="9f6a9172-5491-4a00-aea2-11d0be8a6e46" containerName="main" containerID="cri-o://e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98" gracePeriod=45
Sep 09 21:58:00.224439 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.224371    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:58:00.334646 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.334606    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tls-certs\") pod \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") "
Sep 09 21:58:00.334868 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.334655    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-dshm\") pod \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") "
Sep 09 21:58:00.334868 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.334683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-home\") pod \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") "
Sep 09 21:58:00.334868 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.334735    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tmp-dir\") pod \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") "
Sep 09 21:58:00.334868 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.334830    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-model-cache\") pod \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") "
Sep 09 21:58:00.335079 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.334905    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw6hk\" (UniqueName: \"kubernetes.io/projected/9f6a9172-5491-4a00-aea2-11d0be8a6e46-kube-api-access-vw6hk\") pod \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\" (UID: \"9f6a9172-5491-4a00-aea2-11d0be8a6e46\") "
Sep 09 21:58:00.335079 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.335014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-home" (OuterVolumeSpecName: "home") pod "9f6a9172-5491-4a00-aea2-11d0be8a6e46" (UID: "9f6a9172-5491-4a00-aea2-11d0be8a6e46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:00.335079 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.335055    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f6a9172-5491-4a00-aea2-11d0be8a6e46" (UID: "9f6a9172-5491-4a00-aea2-11d0be8a6e46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:00.335239 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.335140    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f6a9172-5491-4a00-aea2-11d0be8a6e46" (UID: "9f6a9172-5491-4a00-aea2-11d0be8a6e46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:00.335282 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.335270    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:00.335352 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.335290    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:00.335352 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.335318    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:00.337359 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.337334    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6a9172-5491-4a00-aea2-11d0be8a6e46-kube-api-access-vw6hk" (OuterVolumeSpecName: "kube-api-access-vw6hk") pod "9f6a9172-5491-4a00-aea2-11d0be8a6e46" (UID: "9f6a9172-5491-4a00-aea2-11d0be8a6e46"). InnerVolumeSpecName "kube-api-access-vw6hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:58:00.337493 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.337474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-dshm" (OuterVolumeSpecName: "dshm") pod "9f6a9172-5491-4a00-aea2-11d0be8a6e46" (UID: "9f6a9172-5491-4a00-aea2-11d0be8a6e46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:00.337555 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.337486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f6a9172-5491-4a00-aea2-11d0be8a6e46" (UID: "9f6a9172-5491-4a00-aea2-11d0be8a6e46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:58:00.359707 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.359675    2563 generic.go:356] "Generic (PLEG): container finished" podID="9f6a9172-5491-4a00-aea2-11d0be8a6e46" containerID="e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98" exitCode=0
Sep 09 21:58:00.359918 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.359730    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"
Sep 09 21:58:00.359918 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.359732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8" event={"ID":"9f6a9172-5491-4a00-aea2-11d0be8a6e46","Type":"ContainerDied","Data":"e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98"}
Sep 09 21:58:00.359918 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.359813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8" event={"ID":"9f6a9172-5491-4a00-aea2-11d0be8a6e46","Type":"ContainerDied","Data":"041ae247a7c976d0c567dc0109a15768f45279fb9df361fd3c281f104573e760"}
Sep 09 21:58:00.359918 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.359829    2563 scope.go:117] "RemoveContainer" containerID="e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98"
Sep 09 21:58:00.369335 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.369316    2563 scope.go:117] "RemoveContainer" containerID="e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98"
Sep 09 21:58:00.369621 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:00.369591    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98\": container with ID starting with e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98 not found: ID does not exist" containerID="e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98"
Sep 09 21:58:00.369668 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.369626    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98"} err="failed to get container status \"e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98\": rpc error: code = NotFound desc = could not find container \"e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98\": container with ID starting with e651a104fe038f1fb25f31cc534c6e3daf66b42df2d226cf3e36ba0ae0cd8b98 not found: ID does not exist"
Sep 09 21:58:00.384245 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.384221    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"]
Sep 09 21:58:00.389101 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.389081    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-bbdd85/explicit-model-bbdd85-kserve-68d4cc99c5-n5sp8"]
Sep 09 21:58:00.436776 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.436747    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw6hk\" (UniqueName: \"kubernetes.io/projected/9f6a9172-5491-4a00-aea2-11d0be8a6e46-kube-api-access-vw6hk\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:00.436920 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.436799    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6a9172-5491-4a00-aea2-11d0be8a6e46-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:00.436920 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:00.436813    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6a9172-5491-4a00-aea2-11d0be8a6e46-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:01.615107 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:01.615073    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f6a9172-5491-4a00-aea2-11d0be8a6e46" path="/var/lib/kubelet/pods/9f6a9172-5491-4a00-aea2-11d0be8a6e46/volumes"
Sep 09 21:58:03.245748 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:03.245711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:03.415946 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:03.415911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:03.655062 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:03.654984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:04.108926 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:04.108891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:04.514066 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:04.514038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:05.568479 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:05.568443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:06.099752 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:06.099719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:06.232657 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:06.232628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:06.353573 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:06.353487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:09.465154 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:09.465124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:15.412206 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:15.412170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:15.623382 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:15.623340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:15.734602 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:15.734521    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:16.111129 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.111096    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"]
Sep 09 21:58:16.111507 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.111495    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6a9172-5491-4a00-aea2-11d0be8a6e46" containerName="main"
Sep 09 21:58:16.111551 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.111509    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6a9172-5491-4a00-aea2-11d0be8a6e46" containerName="main"
Sep 09 21:58:16.111551 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.111525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a617c8-b3fa-4a80-85b3-a8b41d82b45a" containerName="maas-api"
Sep 09 21:58:16.111551 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.111530    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a617c8-b3fa-4a80-85b3-a8b41d82b45a" containerName="maas-api"
Sep 09 21:58:16.111648 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.111601    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6a9172-5491-4a00-aea2-11d0be8a6e46" containerName="main"
Sep 09 21:58:16.111648 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.111611    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a617c8-b3fa-4a80-85b3-a8b41d82b45a" containerName="maas-api"
Sep 09 21:58:16.116149 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.116132    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.121599 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.121579    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c10ca1\"/\"kube-root-ca.crt\""
Sep 09 21:58:16.122754 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.122732    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c10ca1\"/\"noten-model-c10ca1-kserve-self-signed-certs\""
Sep 09 21:58:16.122754 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.122746    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c10ca1\"/\"default-dockercfg-tb4tw\""
Sep 09 21:58:16.123220 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.122743    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c10ca1\"/\"openshift-service-ca.crt\""
Sep 09 21:58:16.144175 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.144144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"]
Sep 09 21:58:16.200917 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.200888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:16.291681 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.291647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djcb\" (UniqueName: \"kubernetes.io/projected/f4185501-25d3-442c-a4c2-44818d747f54-kube-api-access-6djcb\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.291880 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.291693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-home\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.291880 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.291794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.291880 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.291828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-tmp-dir\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.291880 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.291847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-dshm\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.292030 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.291933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-model-cache\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.392880 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.392738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-model-cache\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.392880 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.392874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djcb\" (UniqueName: \"kubernetes.io/projected/f4185501-25d3-442c-a4c2-44818d747f54-kube-api-access-6djcb\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.393138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.392906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-home\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.393138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.392944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.393138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.392970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-tmp-dir\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.393138 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.393089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-dshm\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.393334 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.393168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-model-cache\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.393334 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.393282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-home\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.393334 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.393303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-tmp-dir\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.395344 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.395310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-dshm\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.395627 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.395606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.407549 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.407524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djcb\" (UniqueName: \"kubernetes.io/projected/f4185501-25d3-442c-a4c2-44818d747f54-kube-api-access-6djcb\") pod \"noten-model-c10ca1-kserve-5d7fcf6587-lxftc\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") " pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.425490 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.425457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:16.725453 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:16.725414    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"]
Sep 09 21:58:16.726780 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:58:16.726738    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4185501_25d3_442c_a4c2_44818d747f54.slice/crio-7e02e33ab43a17483c7451774322d3de180977db6b4d6007ee487bc524c2fc7c WatchSource:0}: Error finding container 7e02e33ab43a17483c7451774322d3de180977db6b4d6007ee487bc524c2fc7c: Status 404 returned error can't find the container with id 7e02e33ab43a17483c7451774322d3de180977db6b4d6007ee487bc524c2fc7c
Sep 09 21:58:17.470784 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:17.470737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc" event={"ID":"f4185501-25d3-442c-a4c2-44818d747f54","Type":"ContainerStarted","Data":"fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b"}
Sep 09 21:58:17.471253 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:17.470794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc" event={"ID":"f4185501-25d3-442c-a4c2-44818d747f54","Type":"ContainerStarted","Data":"7e02e33ab43a17483c7451774322d3de180977db6b4d6007ee487bc524c2fc7c"}
Sep 09 21:58:18.328246 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:18.328190    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc" podStartSLOduration=2.328172497 podStartE2EDuration="2.328172497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:58:17.501332701 +0000 UTC m=+1190.397998708" watchObservedRunningTime="2026-09-09 21:58:18.328172497 +0000 UTC m=+1191.224838505"
Sep 09 21:58:18.328466 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:18.328452    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"]
Sep 09 21:58:18.394713 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:18.394680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:18.574150 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:18.574114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:18.616530 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:18.616446    2563 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-self-signed-certs: secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:18.616530 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:18.616524    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs podName:f4185501-25d3-442c-a4c2-44818d747f54 nodeName:}" failed. No retries permitted until 2026-09-09 21:58:19.11650331 +0000 UTC m=+1192.013169295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs") pod "noten-model-c10ca1-kserve-5d7fcf6587-lxftc" (UID: "f4185501-25d3-442c-a4c2-44818d747f54") : secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:18.809474 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:18.809442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:18.970405 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:18.970318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:19.119308 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:19.119273    2563 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-self-signed-certs: secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:19.119491 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:19.119342    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs podName:f4185501-25d3-442c-a4c2-44818d747f54 nodeName:}" failed. No retries permitted until 2026-09-09 21:58:20.119328868 +0000 UTC m=+1193.015994853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs") pod "noten-model-c10ca1-kserve-5d7fcf6587-lxftc" (UID: "f4185501-25d3-442c-a4c2-44818d747f54") : secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:20.128421 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:20.128382    2563 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-self-signed-certs: secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:20.128852 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:20.128457    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs podName:f4185501-25d3-442c-a4c2-44818d747f54 nodeName:}" failed. No retries permitted until 2026-09-09 21:58:22.128441815 +0000 UTC m=+1195.025107805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs") pod "noten-model-c10ca1-kserve-5d7fcf6587-lxftc" (UID: "f4185501-25d3-442c-a4c2-44818d747f54") : secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:22.143818 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:22.143758    2563 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-self-signed-certs: secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:22.144204 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:22.143854    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs podName:f4185501-25d3-442c-a4c2-44818d747f54 nodeName:}" failed. No retries permitted until 2026-09-09 21:58:26.143840268 +0000 UTC m=+1199.040506253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs") pod "noten-model-c10ca1-kserve-5d7fcf6587-lxftc" (UID: "f4185501-25d3-442c-a4c2-44818d747f54") : secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:23.879592 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:23.879551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:26.173516 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:26.173481    2563 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-self-signed-certs: secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:26.173970 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:26.173550    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs podName:f4185501-25d3-442c-a4c2-44818d747f54 nodeName:}" failed. No retries permitted until 2026-09-09 21:58:34.173536234 +0000 UTC m=+1207.070202220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs") pod "noten-model-c10ca1-kserve-5d7fcf6587-lxftc" (UID: "f4185501-25d3-442c-a4c2-44818d747f54") : secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:26.426294 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:26.426203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:27.613552 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:27.613527    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:58:27.619256 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:27.619233    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 21:58:31.534282 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:31.534237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:34.243071 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:34.243034    2563 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-self-signed-certs: secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:34.243498 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:34.243122    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs podName:f4185501-25d3-442c-a4c2-44818d747f54 nodeName:}" failed. No retries permitted until 2026-09-09 21:58:50.243106241 +0000 UTC m=+1223.139772227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs") pod "noten-model-c10ca1-kserve-5d7fcf6587-lxftc" (UID: "f4185501-25d3-442c-a4c2-44818d747f54") : secret "noten-model-c10ca1-kserve-self-signed-certs" not found
Sep 09 21:58:34.504397 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.504359    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc" podUID="f4185501-25d3-442c-a4c2-44818d747f54" containerName="main" containerID="cri-o://fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b" gracePeriod=45
Sep 09 21:58:34.758457 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.758399    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:34.848637 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.848609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-tmp-dir\") pod \"f4185501-25d3-442c-a4c2-44818d747f54\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") "
Sep 09 21:58:34.848822 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.848666    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6djcb\" (UniqueName: \"kubernetes.io/projected/f4185501-25d3-442c-a4c2-44818d747f54-kube-api-access-6djcb\") pod \"f4185501-25d3-442c-a4c2-44818d747f54\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") "
Sep 09 21:58:34.848822 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.848683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-dshm\") pod \"f4185501-25d3-442c-a4c2-44818d747f54\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") "
Sep 09 21:58:34.848822 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.848796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-model-cache\") pod \"f4185501-25d3-442c-a4c2-44818d747f54\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") "
Sep 09 21:58:34.848996 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.848841    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-home\") pod \"f4185501-25d3-442c-a4c2-44818d747f54\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") "
Sep 09 21:58:34.848996 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.848901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs\") pod \"f4185501-25d3-442c-a4c2-44818d747f54\" (UID: \"f4185501-25d3-442c-a4c2-44818d747f54\") "
Sep 09 21:58:34.848996 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.848927    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4185501-25d3-442c-a4c2-44818d747f54" (UID: "f4185501-25d3-442c-a4c2-44818d747f54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:34.849146 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.849092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4185501-25d3-442c-a4c2-44818d747f54" (UID: "f4185501-25d3-442c-a4c2-44818d747f54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:34.849146 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.849130    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-home" (OuterVolumeSpecName: "home") pod "f4185501-25d3-442c-a4c2-44818d747f54" (UID: "f4185501-25d3-442c-a4c2-44818d747f54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:34.849266 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.849247    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:34.849266 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.849265    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:34.849422 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.849273    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:34.851116 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.851093    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4185501-25d3-442c-a4c2-44818d747f54" (UID: "f4185501-25d3-442c-a4c2-44818d747f54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:58:34.851428 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.851409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-dshm" (OuterVolumeSpecName: "dshm") pod "f4185501-25d3-442c-a4c2-44818d747f54" (UID: "f4185501-25d3-442c-a4c2-44818d747f54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:34.851489 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.851436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4185501-25d3-442c-a4c2-44818d747f54-kube-api-access-6djcb" (OuterVolumeSpecName: "kube-api-access-6djcb") pod "f4185501-25d3-442c-a4c2-44818d747f54" (UID: "f4185501-25d3-442c-a4c2-44818d747f54"). InnerVolumeSpecName "kube-api-access-6djcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:58:34.950353 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.950310    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4185501-25d3-442c-a4c2-44818d747f54-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:34.950353 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.950346    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6djcb\" (UniqueName: \"kubernetes.io/projected/f4185501-25d3-442c-a4c2-44818d747f54-kube-api-access-6djcb\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:34.950353 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:34.950358    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4185501-25d3-442c-a4c2-44818d747f54-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:35.577602 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:35.577560    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4185501-25d3-442c-a4c2-44818d747f54" containerID="fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b" exitCode=0
Sep 09 21:58:35.578095 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:35.577635    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"
Sep 09 21:58:35.578095 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:35.577655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc" event={"ID":"f4185501-25d3-442c-a4c2-44818d747f54","Type":"ContainerDied","Data":"fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b"}
Sep 09 21:58:35.578095 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:35.577703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc" event={"ID":"f4185501-25d3-442c-a4c2-44818d747f54","Type":"ContainerDied","Data":"7e02e33ab43a17483c7451774322d3de180977db6b4d6007ee487bc524c2fc7c"}
Sep 09 21:58:35.578095 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:35.577721    2563 scope.go:117] "RemoveContainer" containerID="fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b"
Sep 09 21:58:35.587555 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:35.587536    2563 scope.go:117] "RemoveContainer" containerID="fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b"
Sep 09 21:58:35.587837 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:35.587819    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b\": container with ID starting with fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b not found: ID does not exist" containerID="fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b"
Sep 09 21:58:35.587889 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:35.587846    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b"} err="failed to get container status \"fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b\": rpc error: code = NotFound desc = could not find container \"fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b\": container with ID starting with fe1f05154d715f14d72d3df5311b202579dab7d83b768de81ec7291c0a2f6e8b not found: ID does not exist"
Sep 09 21:58:35.606369 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:35.606336    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"]
Sep 09 21:58:35.606369 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:35.606373    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c10ca1/noten-model-c10ca1-kserve-5d7fcf6587-lxftc"]
Sep 09 21:58:37.338507 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:37.338469    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"]
Sep 09 21:58:37.395203 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:37.395167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:37.608141 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:37.608060    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4185501-25d3-442c-a4c2-44818d747f54" path="/var/lib/kubelet/pods/f4185501-25d3-442c-a4c2-44818d747f54/volumes"
Sep 09 21:58:38.235310 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:38.235270    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"]
Sep 09 21:58:38.254664 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:38.254629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:38.486535 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:38.486441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:38.673668 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:38.673634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:38.783164 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:38.783135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:38.878461 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:38.878426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:48.894913 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:48.894879    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"]
Sep 09 21:58:48.895449 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:48.895116    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95" podUID="1653c335-347e-442d-9191-5b650bfa4d46" containerName="maas-api" containerID="cri-o://cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46" gracePeriod=30
Sep 09 21:58:48.940612 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:48.940575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:49.083970 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.083928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:49.130110 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.130071    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95" podUID="1653c335-347e-442d-9191-5b650bfa4d46" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.54:8443/health\": dial tcp 10.132.0.54:8443: connect: connection refused"
Sep 09 21:58:49.172057 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.171966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:49.243911 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.243877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:49.309968 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.309932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:49.354751 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.354728    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:58:49.376913 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.376885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:49.490355 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.490265    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-metrics-certs\") pod \"1653c335-347e-442d-9191-5b650bfa4d46\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") "
Sep 09 21:58:49.490355 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.490315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-maas-api-tls\") pod \"1653c335-347e-442d-9191-5b650bfa4d46\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") "
Sep 09 21:58:49.490597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.490385    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1653c335-347e-442d-9191-5b650bfa4d46-openshift-service-ca\") pod \"1653c335-347e-442d-9191-5b650bfa4d46\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") "
Sep 09 21:58:49.490597 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.490410    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls74g\" (UniqueName: \"kubernetes.io/projected/1653c335-347e-442d-9191-5b650bfa4d46-kube-api-access-ls74g\") pod \"1653c335-347e-442d-9191-5b650bfa4d46\" (UID: \"1653c335-347e-442d-9191-5b650bfa4d46\") "
Sep 09 21:58:49.490755 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.490727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1653c335-347e-442d-9191-5b650bfa4d46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1653c335-347e-442d-9191-5b650bfa4d46" (UID: "1653c335-347e-442d-9191-5b650bfa4d46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:58:49.492682 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.492649    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1653c335-347e-442d-9191-5b650bfa4d46" (UID: "1653c335-347e-442d-9191-5b650bfa4d46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:58:49.492867 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.492695    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1653c335-347e-442d-9191-5b650bfa4d46" (UID: "1653c335-347e-442d-9191-5b650bfa4d46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:58:49.492867 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.492800    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1653c335-347e-442d-9191-5b650bfa4d46-kube-api-access-ls74g" (OuterVolumeSpecName: "kube-api-access-ls74g") pod "1653c335-347e-442d-9191-5b650bfa4d46" (UID: "1653c335-347e-442d-9191-5b650bfa4d46"). InnerVolumeSpecName "kube-api-access-ls74g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:58:49.591272 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.591236    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1653c335-347e-442d-9191-5b650bfa4d46-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:49.591272 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.591263    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls74g\" (UniqueName: \"kubernetes.io/projected/1653c335-347e-442d-9191-5b650bfa4d46-kube-api-access-ls74g\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:49.591272 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.591273    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:49.591272 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.591283    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1653c335-347e-442d-9191-5b650bfa4d46-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:49.594387 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.594365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:49.662568 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.662529    2563 generic.go:356] "Generic (PLEG): container finished" podID="1653c335-347e-442d-9191-5b650bfa4d46" containerID="cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46" exitCode=0
Sep 09 21:58:49.662726 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.662578    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"
Sep 09 21:58:49.662726 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.662609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95" event={"ID":"1653c335-347e-442d-9191-5b650bfa4d46","Type":"ContainerDied","Data":"cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46"}
Sep 09 21:58:49.662726 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.662658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95" event={"ID":"1653c335-347e-442d-9191-5b650bfa4d46","Type":"ContainerDied","Data":"fbfeb532517566a91b034f81274406ce9814d92bfd6582cb441d09140e6cbf31"}
Sep 09 21:58:49.662726 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.662681    2563 scope.go:117] "RemoveContainer" containerID="cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46"
Sep 09 21:58:49.672123 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.672107    2563 scope.go:117] "RemoveContainer" containerID="cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46"
Sep 09 21:58:49.672384 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:49.672366    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46\": container with ID starting with cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46 not found: ID does not exist" containerID="cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46"
Sep 09 21:58:49.672431 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.672392    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46"} err="failed to get container status \"cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46\": rpc error: code = NotFound desc = could not find container \"cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46\": container with ID starting with cf575ba507a1afd17dc75eec41ae05a4d403233996c6c6e379fabe0a16280e46 not found: ID does not exist"
Sep 09 21:58:49.681918 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.681897    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"]
Sep 09 21:58:49.688280 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:49.688257    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5aaa80-56dd4b4698-lcw95"]
Sep 09 21:58:50.161191 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:50.161159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:51.606545 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:51.606517    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1653c335-347e-442d-9191-5b650bfa4d46" path="/var/lib/kubelet/pods/1653c335-347e-442d-9191-5b650bfa4d46/volumes"
Sep 09 21:58:52.330525 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.330494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:52.358448 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.358412    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8" podUID="d8e3ae77-b83d-4d36-843f-7b6a36445ec2" containerName="main" containerID="cri-o://f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e" gracePeriod=45
Sep 09 21:58:52.614814 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.614763    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:58:52.681401 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.681366    2563 generic.go:356] "Generic (PLEG): container finished" podID="d8e3ae77-b83d-4d36-843f-7b6a36445ec2" containerID="f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e" exitCode=0
Sep 09 21:58:52.681567 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.681428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"
Sep 09 21:58:52.681567 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.681449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8" event={"ID":"d8e3ae77-b83d-4d36-843f-7b6a36445ec2","Type":"ContainerDied","Data":"f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e"}
Sep 09 21:58:52.681567 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.681499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8" event={"ID":"d8e3ae77-b83d-4d36-843f-7b6a36445ec2","Type":"ContainerDied","Data":"a9cbdce282d3f196778583418a7769007c509dbaba1dbbbe034b9847dd7ddf22"}
Sep 09 21:58:52.681567 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.681522    2563 scope.go:117] "RemoveContainer" containerID="f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e"
Sep 09 21:58:52.690823 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.690804    2563 scope.go:117] "RemoveContainer" containerID="f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e"
Sep 09 21:58:52.691124 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:52.691103    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e\": container with ID starting with f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e not found: ID does not exist" containerID="f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e"
Sep 09 21:58:52.691226 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.691129    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e"} err="failed to get container status \"f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e\": rpc error: code = NotFound desc = could not find container \"f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e\": container with ID starting with f876d9716da4ff2ba91882d49ca5213577cd019c6b8856557c3b4542d332ee1e not found: ID does not exist"
Sep 09 21:58:52.718727 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.718703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4hfw\" (UniqueName: \"kubernetes.io/projected/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-kube-api-access-m4hfw\") pod \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") "
Sep 09 21:58:52.718875 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.718733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-home\") pod \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") "
Sep 09 21:58:52.718875 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.718755    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tmp-dir\") pod \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") "
Sep 09 21:58:52.718875 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.718813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-dshm\") pod \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") "
Sep 09 21:58:52.719044 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.718914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-model-cache\") pod \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") "
Sep 09 21:58:52.719044 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.718955    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tls-certs\") pod \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\" (UID: \"d8e3ae77-b83d-4d36-843f-7b6a36445ec2\") "
Sep 09 21:58:52.719044 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.719027    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-home" (OuterVolumeSpecName: "home") pod "d8e3ae77-b83d-4d36-843f-7b6a36445ec2" (UID: "d8e3ae77-b83d-4d36-843f-7b6a36445ec2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:52.719181 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.719075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8e3ae77-b83d-4d36-843f-7b6a36445ec2" (UID: "d8e3ae77-b83d-4d36-843f-7b6a36445ec2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:52.719252 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.719223    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8e3ae77-b83d-4d36-843f-7b6a36445ec2" (UID: "d8e3ae77-b83d-4d36-843f-7b6a36445ec2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:52.719396 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.719373    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:52.719474 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.719402    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:52.719474 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.719415    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:52.721188 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.721166    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-kube-api-access-m4hfw" (OuterVolumeSpecName: "kube-api-access-m4hfw") pod "d8e3ae77-b83d-4d36-843f-7b6a36445ec2" (UID: "d8e3ae77-b83d-4d36-843f-7b6a36445ec2"). InnerVolumeSpecName "kube-api-access-m4hfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:58:52.721282 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.721218    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-dshm" (OuterVolumeSpecName: "dshm") pod "d8e3ae77-b83d-4d36-843f-7b6a36445ec2" (UID: "d8e3ae77-b83d-4d36-843f-7b6a36445ec2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:52.721282 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.721256    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8e3ae77-b83d-4d36-843f-7b6a36445ec2" (UID: "d8e3ae77-b83d-4d36-843f-7b6a36445ec2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:58:52.820709 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.820674    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4hfw\" (UniqueName: \"kubernetes.io/projected/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-kube-api-access-m4hfw\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:52.820709 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.820707    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:52.820921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:52.820720    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e3ae77-b83d-4d36-843f-7b6a36445ec2-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:53.005299 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.005263    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"]
Sep 09 21:58:53.011671 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.011643    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5aaa80/test-model-5aaa80-kserve-7d46b9dccc-wplm8"]
Sep 09 21:58:53.255700 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.255656    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv" podUID="fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" containerName="main" containerID="cri-o://3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9" gracePeriod=45
Sep 09 21:58:53.515837 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.515721    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:58:53.607199 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.607171    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8e3ae77-b83d-4d36-843f-7b6a36445ec2" path="/var/lib/kubelet/pods/d8e3ae77-b83d-4d36-843f-7b6a36445ec2/volumes"
Sep 09 21:58:53.627633 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.627590    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tmp-dir\") pod \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") "
Sep 09 21:58:53.627633 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.627634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-home\") pod \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") "
Sep 09 21:58:53.628161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.627657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-model-cache\") pod \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") "
Sep 09 21:58:53.628161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.627685    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-dshm\") pod \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") "
Sep 09 21:58:53.628161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.627714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s26wq\" (UniqueName: \"kubernetes.io/projected/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-kube-api-access-s26wq\") pod \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") "
Sep 09 21:58:53.628161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.627757    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tls-certs\") pod \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\" (UID: \"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad\") "
Sep 09 21:58:53.628161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.627906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" (UID: "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:53.628161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.627925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" (UID: "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:53.628161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.627937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-home" (OuterVolumeSpecName: "home") pod "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" (UID: "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:53.628161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.628053    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:53.628161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.628064    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:53.628161 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.628073    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:53.630204 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.630180    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-dshm" (OuterVolumeSpecName: "dshm") pod "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" (UID: "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:58:53.630204 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.630184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" (UID: "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:58:53.630204 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.630178    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-kube-api-access-s26wq" (OuterVolumeSpecName: "kube-api-access-s26wq") pod "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" (UID: "fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad"). InnerVolumeSpecName "kube-api-access-s26wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:58:53.688874 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.688843    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" containerID="3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9" exitCode=0
Sep 09 21:58:53.689037 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.688886    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"
Sep 09 21:58:53.689037 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.688931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv" event={"ID":"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad","Type":"ContainerDied","Data":"3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9"}
Sep 09 21:58:53.689037 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.688975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv" event={"ID":"fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad","Type":"ContainerDied","Data":"0361783c596b1d27488638464ccb4f5edc785711451966f7a80eb66b87203d50"}
Sep 09 21:58:53.689037 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.688990    2563 scope.go:117] "RemoveContainer" containerID="3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9"
Sep 09 21:58:53.699332 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.699314    2563 scope.go:117] "RemoveContainer" containerID="3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9"
Sep 09 21:58:53.699601 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:58:53.699583    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9\": container with ID starting with 3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9 not found: ID does not exist" containerID="3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9"
Sep 09 21:58:53.699650 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.699611    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9"} err="failed to get container status \"3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9\": rpc error: code = NotFound desc = could not find container \"3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9\": container with ID starting with 3b74f118f428bc6edfee52fc4d709d7a0945e3a083ac776640875bb60a9f10d9 not found: ID does not exist"
Sep 09 21:58:53.714567 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.714535    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"]
Sep 09 21:58:53.724464 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.724433    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ebe9b0/test-model-ebe9b0-kserve-69d484598c-5bgkv"]
Sep 09 21:58:53.729391 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.729367    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:53.729391 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.729389    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:53.729531 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:53.729398    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s26wq\" (UniqueName: \"kubernetes.io/projected/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad-kube-api-access-s26wq\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:58:55.606660 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:55.606630    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" path="/var/lib/kubelet/pods/fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad/volumes"
Sep 09 21:58:55.747031 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:55.747001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:55.935612 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:55.935534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:56.068023 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:56.067989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:56.602166 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:56.602131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:56.768172 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:56.768133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:56.885266 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:56.885163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:57.007457 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:57.007424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:57.100099 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:57.100059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:57.761345 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:57.761311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:57.848653 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:57.848618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:59.388635 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:58:59.388586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:01.025750 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:01.025713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:05.399310 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:05.399274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:05.508628 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:05.508586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:06.070067 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:06.069999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:07.207315 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.207273    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"]
Sep 09 21:59:07.207817 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.207563    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb" podUID="272d003c-cb92-4bb9-88b0-b06a6d60dddb" containerName="maas-api" containerID="cri-o://0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b" gracePeriod=30
Sep 09 21:59:07.264740 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.264714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:07.334719 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.334683    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"]
Sep 09 21:59:07.335013 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.334966    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4" podUID="abc200bc-76b9-4695-b496-c4d42ea37300" containerName="payload-pre-processing" containerID="cri-o://2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c" gracePeriod=30
Sep 09 21:59:07.386170 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.386130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:07.470228 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.470165    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:59:07.487660 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.487631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:07.564969 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.564935    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-metrics-certs\") pod \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") "
Sep 09 21:59:07.565155 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.564995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-maas-api-tls\") pod \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") "
Sep 09 21:59:07.565155 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.565070    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z7qt\" (UniqueName: \"kubernetes.io/projected/272d003c-cb92-4bb9-88b0-b06a6d60dddb-kube-api-access-2z7qt\") pod \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") "
Sep 09 21:59:07.565155 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.565095    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272d003c-cb92-4bb9-88b0-b06a6d60dddb-openshift-service-ca\") pod \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\" (UID: \"272d003c-cb92-4bb9-88b0-b06a6d60dddb\") "
Sep 09 21:59:07.565544 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.565514    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272d003c-cb92-4bb9-88b0-b06a6d60dddb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "272d003c-cb92-4bb9-88b0-b06a6d60dddb" (UID: "272d003c-cb92-4bb9-88b0-b06a6d60dddb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:07.567297 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.567267    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272d003c-cb92-4bb9-88b0-b06a6d60dddb-kube-api-access-2z7qt" (OuterVolumeSpecName: "kube-api-access-2z7qt") pod "272d003c-cb92-4bb9-88b0-b06a6d60dddb" (UID: "272d003c-cb92-4bb9-88b0-b06a6d60dddb"). InnerVolumeSpecName "kube-api-access-2z7qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:07.567407 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.567364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "272d003c-cb92-4bb9-88b0-b06a6d60dddb" (UID: "272d003c-cb92-4bb9-88b0-b06a6d60dddb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:59:07.567407 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.567382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "272d003c-cb92-4bb9-88b0-b06a6d60dddb" (UID: "272d003c-cb92-4bb9-88b0-b06a6d60dddb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:59:07.594268 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.594196    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:07.606121 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.606092    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:59:07.666417 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.666387    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfhr4\" (UniqueName: \"kubernetes.io/projected/abc200bc-76b9-4695-b496-c4d42ea37300-kube-api-access-mfhr4\") pod \"abc200bc-76b9-4695-b496-c4d42ea37300\" (UID: \"abc200bc-76b9-4695-b496-c4d42ea37300\") "
Sep 09 21:59:07.666605 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.666460    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc200bc-76b9-4695-b496-c4d42ea37300-plugins-config-volume\") pod \"abc200bc-76b9-4695-b496-c4d42ea37300\" (UID: \"abc200bc-76b9-4695-b496-c4d42ea37300\") "
Sep 09 21:59:07.666605 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.666602    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:59:07.666731 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.666612    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272d003c-cb92-4bb9-88b0-b06a6d60dddb-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:59:07.666731 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.666620    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7qt\" (UniqueName: \"kubernetes.io/projected/272d003c-cb92-4bb9-88b0-b06a6d60dddb-kube-api-access-2z7qt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:59:07.666731 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.666630    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272d003c-cb92-4bb9-88b0-b06a6d60dddb-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:59:07.666969 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.666938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc200bc-76b9-4695-b496-c4d42ea37300-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abc200bc-76b9-4695-b496-c4d42ea37300" (UID: "abc200bc-76b9-4695-b496-c4d42ea37300"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:07.668882 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.668858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc200bc-76b9-4695-b496-c4d42ea37300-kube-api-access-mfhr4" (OuterVolumeSpecName: "kube-api-access-mfhr4") pod "abc200bc-76b9-4695-b496-c4d42ea37300" (UID: "abc200bc-76b9-4695-b496-c4d42ea37300"). InnerVolumeSpecName "kube-api-access-mfhr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:07.708835 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.708799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:07.767816 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.767744    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfhr4\" (UniqueName: \"kubernetes.io/projected/abc200bc-76b9-4695-b496-c4d42ea37300-kube-api-access-mfhr4\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:59:07.767816 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.767815    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc200bc-76b9-4695-b496-c4d42ea37300-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:59:07.775433 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.775406    2563 generic.go:356] "Generic (PLEG): container finished" podID="abc200bc-76b9-4695-b496-c4d42ea37300" containerID="2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c" exitCode=0
Sep 09 21:59:07.775576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.775441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4" event={"ID":"abc200bc-76b9-4695-b496-c4d42ea37300","Type":"ContainerDied","Data":"2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c"}
Sep 09 21:59:07.775576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.775493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4" event={"ID":"abc200bc-76b9-4695-b496-c4d42ea37300","Type":"ContainerDied","Data":"44e96f5cff458c6a8410e63983a706778a66558436677847dc79c71314fdeecd"}
Sep 09 21:59:07.775576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.775455    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"
Sep 09 21:59:07.775576 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.775537    2563 scope.go:117] "RemoveContainer" containerID="2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c"
Sep 09 21:59:07.776888 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.776864    2563 generic.go:356] "Generic (PLEG): container finished" podID="272d003c-cb92-4bb9-88b0-b06a6d60dddb" containerID="0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b" exitCode=0
Sep 09 21:59:07.777016 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.776905    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"
Sep 09 21:59:07.777016 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.776945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb" event={"ID":"272d003c-cb92-4bb9-88b0-b06a6d60dddb","Type":"ContainerDied","Data":"0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b"}
Sep 09 21:59:07.777016 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.777000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb" event={"ID":"272d003c-cb92-4bb9-88b0-b06a6d60dddb","Type":"ContainerDied","Data":"3b2aa63419cc02ac83fb1b6bdb3b44913683b6da7159a7ceb2ec67c98d007ead"}
Sep 09 21:59:07.785280 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.785263    2563 scope.go:117] "RemoveContainer" containerID="2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c"
Sep 09 21:59:07.785577 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:59:07.785553    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c\": container with ID starting with 2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c not found: ID does not exist" containerID="2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c"
Sep 09 21:59:07.785612 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.785587    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c"} err="failed to get container status \"2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c\": rpc error: code = NotFound desc = could not find container \"2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c\": container with ID starting with 2199ff723300f679c2b1a3caf1bbe723d7ae2ded31f2fbe2b5800b29047ec05c not found: ID does not exist"
Sep 09 21:59:07.785612 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.785605    2563 scope.go:117] "RemoveContainer" containerID="0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b"
Sep 09 21:59:07.793906 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.793889    2563 scope.go:117] "RemoveContainer" containerID="0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b"
Sep 09 21:59:07.795823 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:59:07.794409    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b\": container with ID starting with 0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b not found: ID does not exist" containerID="0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b"
Sep 09 21:59:07.795823 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.794439    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b"} err="failed to get container status \"0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b\": rpc error: code = NotFound desc = could not find container \"0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b\": container with ID starting with 0e68ac13771e033bf09c1a05c7af6fb990cb6d6aa5bdd0d5ec6006f959e1c27b not found: ID does not exist"
Sep 09 21:59:07.801846 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.801825    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"]
Sep 09 21:59:07.807947 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.807928    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ebe9b0-77d9f45794-p4hgb"]
Sep 09 21:59:07.826022 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.826001    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"]
Sep 09 21:59:07.830921 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.830902    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ebe9b0-7784dc5b74-26jt4"]
Sep 09 21:59:07.959712 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:07.959683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:09.605843 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:09.605807    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272d003c-cb92-4bb9-88b0-b06a6d60dddb" path="/var/lib/kubelet/pods/272d003c-cb92-4bb9-88b0-b06a6d60dddb/volumes"
Sep 09 21:59:09.606343 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:09.606318    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc200bc-76b9-4695-b496-c4d42ea37300" path="/var/lib/kubelet/pods/abc200bc-76b9-4695-b496-c4d42ea37300/volumes"
Sep 09 21:59:13.026969 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:13.026937    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:13.126650 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:13.126620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:13.270622 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:13.270582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:13.589757 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:13.589721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:13.879253 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:13.879172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:15.745532 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:15.745503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:15.849486 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:15.849446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:16.054127 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:16.054038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:16.672545 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:16.672510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:17.069832 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:17.069797    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:17.268494 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:17.268458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:17.486233 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:17.486150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:18.219598 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.219570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:18.320419 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.320384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:18.421563 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.421533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:18.765561 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.765484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:18.858291 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.858260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:18.921847 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.921814    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"]
Sep 09 21:59:18.922262 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922245    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc200bc-76b9-4695-b496-c4d42ea37300" containerName="payload-pre-processing"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922265    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc200bc-76b9-4695-b496-c4d42ea37300" containerName="payload-pre-processing"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922292    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272d003c-cb92-4bb9-88b0-b06a6d60dddb" containerName="maas-api"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922301    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="272d003c-cb92-4bb9-88b0-b06a6d60dddb" containerName="maas-api"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922312    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1653c335-347e-442d-9191-5b650bfa4d46" containerName="maas-api"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922320    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1653c335-347e-442d-9191-5b650bfa4d46" containerName="maas-api"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922337    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" containerName="main"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922345    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" containerName="main"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922359    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4185501-25d3-442c-a4c2-44818d747f54" containerName="main"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922366    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4185501-25d3-442c-a4c2-44818d747f54" containerName="main"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922374    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8e3ae77-b83d-4d36-843f-7b6a36445ec2" containerName="main"
Sep 09 21:59:18.922378 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922382    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8e3ae77-b83d-4d36-843f-7b6a36445ec2" containerName="main"
Sep 09 21:59:18.922914 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922508    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1653c335-347e-442d-9191-5b650bfa4d46" containerName="maas-api"
Sep 09 21:59:18.922914 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922520    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe79d8aa-80e1-4bcd-9ecc-ee120d7560ad" containerName="main"
Sep 09 21:59:18.922914 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922534    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="272d003c-cb92-4bb9-88b0-b06a6d60dddb" containerName="maas-api"
Sep 09 21:59:18.922914 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922545    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4185501-25d3-442c-a4c2-44818d747f54" containerName="main"
Sep 09 21:59:18.922914 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922555    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8e3ae77-b83d-4d36-843f-7b6a36445ec2" containerName="main"
Sep 09 21:59:18.922914 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.922562    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc200bc-76b9-4695-b496-c4d42ea37300" containerName="payload-pre-processing"
Sep 09 21:59:18.927205 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.927183    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 21:59:18.931090 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.931065    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-df8f95ff-dockercfg-tfzn8\""
Sep 09 21:59:18.931201 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.931134    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-df8f95ff\""
Sep 09 21:59:18.937794 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.937756    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"]
Sep 09 21:59:18.974869 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.974838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-plugins-config-volume\") pod \"payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9\" (UID: \"3ef82d2b-d21b-41e5-841e-9a44a8b177a0\") " pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 21:59:18.975024 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:18.974875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfnx2\" (UniqueName: \"kubernetes.io/projected/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-kube-api-access-cfnx2\") pod \"payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9\" (UID: \"3ef82d2b-d21b-41e5-841e-9a44a8b177a0\") " pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 21:59:19.076228 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:19.076136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-plugins-config-volume\") pod \"payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9\" (UID: \"3ef82d2b-d21b-41e5-841e-9a44a8b177a0\") " pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 21:59:19.076228 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:19.076177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfnx2\" (UniqueName: \"kubernetes.io/projected/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-kube-api-access-cfnx2\") pod \"payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9\" (UID: \"3ef82d2b-d21b-41e5-841e-9a44a8b177a0\") " pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 21:59:19.076875 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:19.076857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-plugins-config-volume\") pod \"payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9\" (UID: \"3ef82d2b-d21b-41e5-841e-9a44a8b177a0\") " pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 21:59:19.085685 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:19.085661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfnx2\" (UniqueName: \"kubernetes.io/projected/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-kube-api-access-cfnx2\") pod \"payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9\" (UID: \"3ef82d2b-d21b-41e5-841e-9a44a8b177a0\") " pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 21:59:19.238176 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:19.238136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 21:59:19.516532 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:19.516508    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"]
Sep 09 21:59:19.518596 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:59:19.518557    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef82d2b_d21b_41e5_841e_9a44a8b177a0.slice/crio-437f5d33d8470b25d01243b0911a268ad0dfcef5dadfe926d27d1629a92b2662 WatchSource:0}: Error finding container 437f5d33d8470b25d01243b0911a268ad0dfcef5dadfe926d27d1629a92b2662: Status 404 returned error can't find the container with id 437f5d33d8470b25d01243b0911a268ad0dfcef5dadfe926d27d1629a92b2662
Sep 09 21:59:19.855802 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:19.855681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9" event={"ID":"3ef82d2b-d21b-41e5-841e-9a44a8b177a0","Type":"ContainerStarted","Data":"b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab"}
Sep 09 21:59:19.855802 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:19.855747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9" event={"ID":"3ef82d2b-d21b-41e5-841e-9a44a8b177a0","Type":"ContainerStarted","Data":"437f5d33d8470b25d01243b0911a268ad0dfcef5dadfe926d27d1629a92b2662"}
Sep 09 21:59:19.856113 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:19.855922    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 21:59:19.915016 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:19.914965    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9" podStartSLOduration=1.914945145 podStartE2EDuration="1.914945145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:19.91378836 +0000 UTC m=+1252.810454378" watchObservedRunningTime="2026-09-09 21:59:19.914945145 +0000 UTC m=+1252.811611194"
Sep 09 21:59:20.098477 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:20.098436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:20.201620 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:20.201513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:21.479310 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:21.479263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:21.575150 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:21.575117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:21.836644 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:21.836614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:22.422715 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.422682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:22.535679 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.535644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:22.699148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.699071    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"]
Sep 09 21:59:22.703085 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.703067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 21:59:22.706742 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.706718    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ace629\""
Sep 09 21:59:22.706941 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.706912    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ace629-dockercfg-wjft7\""
Sep 09 21:59:22.717745 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.717721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"]
Sep 09 21:59:22.809963 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.809927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/423c2f38-05fc-494d-8468-9dd124dc7c59-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj\" (UID: \"423c2f38-05fc-494d-8468-9dd124dc7c59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 21:59:22.810174 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.809976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq69m\" (UniqueName: \"kubernetes.io/projected/423c2f38-05fc-494d-8468-9dd124dc7c59-kube-api-access-dq69m\") pod \"payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj\" (UID: \"423c2f38-05fc-494d-8468-9dd124dc7c59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 21:59:22.910902 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.910869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/423c2f38-05fc-494d-8468-9dd124dc7c59-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj\" (UID: \"423c2f38-05fc-494d-8468-9dd124dc7c59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 21:59:22.911090 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.910912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq69m\" (UniqueName: \"kubernetes.io/projected/423c2f38-05fc-494d-8468-9dd124dc7c59-kube-api-access-dq69m\") pod \"payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj\" (UID: \"423c2f38-05fc-494d-8468-9dd124dc7c59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 21:59:22.911564 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.911543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/423c2f38-05fc-494d-8468-9dd124dc7c59-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj\" (UID: \"423c2f38-05fc-494d-8468-9dd124dc7c59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 21:59:22.921958 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:22.921932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq69m\" (UniqueName: \"kubernetes.io/projected/423c2f38-05fc-494d-8468-9dd124dc7c59-kube-api-access-dq69m\") pod \"payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj\" (UID: \"423c2f38-05fc-494d-8468-9dd124dc7c59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 21:59:23.013505 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:23.013459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 21:59:23.298621 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:23.298592    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"]
Sep 09 21:59:23.300530 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:59:23.300506    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423c2f38_05fc_494d_8468_9dd124dc7c59.slice/crio-eb7d1fd2bd1020499a0a76c7cc42a196df74c7b0a10419839e101a11e28eb85d WatchSource:0}: Error finding container eb7d1fd2bd1020499a0a76c7cc42a196df74c7b0a10419839e101a11e28eb85d: Status 404 returned error can't find the container with id eb7d1fd2bd1020499a0a76c7cc42a196df74c7b0a10419839e101a11e28eb85d
Sep 09 21:59:23.880724 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:23.880691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj" event={"ID":"423c2f38-05fc-494d-8468-9dd124dc7c59","Type":"ContainerStarted","Data":"700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a"}
Sep 09 21:59:23.881148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:23.880746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj" event={"ID":"423c2f38-05fc-494d-8468-9dd124dc7c59","Type":"ContainerStarted","Data":"eb7d1fd2bd1020499a0a76c7cc42a196df74c7b0a10419839e101a11e28eb85d"}
Sep 09 21:59:23.881148 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:23.880797    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 21:59:23.904964 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:23.904917    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj" podStartSLOduration=1.904902211 podStartE2EDuration="1.904902211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:23.901398924 +0000 UTC m=+1256.798064931" watchObservedRunningTime="2026-09-09 21:59:23.904902211 +0000 UTC m=+1256.801568218"
Sep 09 21:59:30.863063 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:30.863028    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 21:59:34.888854 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:34.888819    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 21:59:39.278171 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:39.278123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:41.907453 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:41.907416    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:42.536942 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:42.536911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:42.638745 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:42.638715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:42.950221 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:42.950144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:43.049654 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:43.049611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:44.925370 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:44.925328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:45.292481 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:45.292455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:45.609303 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:45.609213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:46.254915 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:46.254878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:49.195705 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:49.195675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:49.290500 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:49.290464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:49.403866 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:49.403833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:49.510423 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:49.510398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:49.970442 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:49.970360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:50.692947 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.692914    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"]
Sep 09 21:59:50.696471 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.696440    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.699120 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.699098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-3b2190-serving-cert\""
Sep 09 21:59:50.712934 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.712911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"]
Sep 09 21:59:50.751485 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.751454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:50.770798 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.770740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr924\" (UniqueName: \"kubernetes.io/projected/71470ad2-0883-4473-8ab3-8c5e963871d1-kube-api-access-qr924\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.770979 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.770810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.770979 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.770836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71470ad2-0883-4473-8ab3-8c5e963871d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.770979 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.770861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.852619 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.852592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:50.872137 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.872099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.872293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.872204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr924\" (UniqueName: \"kubernetes.io/projected/71470ad2-0883-4473-8ab3-8c5e963871d1-kube-api-access-qr924\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.872293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.872245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.872293 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.872280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71470ad2-0883-4473-8ab3-8c5e963871d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.872458 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:59:50.872381    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-serving-cert: secret "maas-api-e2e-shared-b-w3-3b2190-serving-cert" not found
Sep 09 21:59:50.872511 ip-10-0-129-45 kubenswrapper[2563]: E0909 21:59:50.872472    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-maas-api-tls podName:71470ad2-0883-4473-8ab3-8c5e963871d1 nodeName:}" failed. No retries permitted until 2026-09-09 21:59:51.372450234 +0000 UTC m=+1284.269116235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-maas-api-tls") pod "maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk" (UID: "71470ad2-0883-4473-8ab3-8c5e963871d1") : secret "maas-api-e2e-shared-b-w3-3b2190-serving-cert" not found
Sep 09 21:59:50.872924 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.872904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71470ad2-0883-4473-8ab3-8c5e963871d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.874716 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.874693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:50.882044 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:50.882018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr924\" (UniqueName: \"kubernetes.io/projected/71470ad2-0883-4473-8ab3-8c5e963871d1-kube-api-access-qr924\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:51.078559 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.078529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:51.302286 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.302243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:51.376175 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.376081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:51.379145 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.379114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:51.392407 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.392377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:51.481554 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.481513    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"]
Sep 09 21:59:51.485364 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.485344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 21:59:51.488050 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.488030    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-3b2190-dockercfg-s4st4\""
Sep 09 21:59:51.488167 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.488071    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-3b2190\""
Sep 09 21:59:51.497801 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.497740    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"]
Sep 09 21:59:51.578198 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.578165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z\" (UID: \"2e96ca93-b449-445e-b2a8-0dd4cb76ee78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 21:59:51.578362 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.578208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x946s\" (UniqueName: \"kubernetes.io/projected/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-kube-api-access-x946s\") pod \"payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z\" (UID: \"2e96ca93-b449-445e-b2a8-0dd4cb76ee78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 21:59:51.609725 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.609696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:51.679833 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.679794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z\" (UID: \"2e96ca93-b449-445e-b2a8-0dd4cb76ee78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 21:59:51.679981 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.679871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x946s\" (UniqueName: \"kubernetes.io/projected/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-kube-api-access-x946s\") pod \"payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z\" (UID: \"2e96ca93-b449-445e-b2a8-0dd4cb76ee78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 21:59:51.680955 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.680926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z\" (UID: \"2e96ca93-b449-445e-b2a8-0dd4cb76ee78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 21:59:51.689658 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.689599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x946s\" (UniqueName: \"kubernetes.io/projected/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-kube-api-access-x946s\") pod \"payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z\" (UID: \"2e96ca93-b449-445e-b2a8-0dd4cb76ee78\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 21:59:51.797211 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.797178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 21:59:51.828343 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.828314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:51.932859 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:59:51.932815    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71470ad2_0883_4473_8ab3_8c5e963871d1.slice/crio-48aba10dd2e151eed6c255cd0be61fb5237e6b32fd29146f6fced44c09f6bc98 WatchSource:0}: Error finding container 48aba10dd2e151eed6c255cd0be61fb5237e6b32fd29146f6fced44c09f6bc98: Status 404 returned error can't find the container with id 48aba10dd2e151eed6c255cd0be61fb5237e6b32fd29146f6fced44c09f6bc98
Sep 09 21:59:51.934062 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.934045    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:59:51.942889 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:51.942867    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"]
Sep 09 21:59:52.039705 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:52.039673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk" event={"ID":"71470ad2-0883-4473-8ab3-8c5e963871d1","Type":"ContainerStarted","Data":"48aba10dd2e151eed6c255cd0be61fb5237e6b32fd29146f6fced44c09f6bc98"}
Sep 09 21:59:52.098546 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:52.098518    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"]
Sep 09 21:59:52.100289 ip-10-0-129-45 kubenswrapper[2563]: W0909 21:59:52.100260    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e96ca93_b449_445e_b2a8_0dd4cb76ee78.slice/crio-5bf65be4a27a0778ce52f7d0a952ed00418b3e372442efb21c7b2cd6b153e5ef WatchSource:0}: Error finding container 5bf65be4a27a0778ce52f7d0a952ed00418b3e372442efb21c7b2cd6b153e5ef: Status 404 returned error can't find the container with id 5bf65be4a27a0778ce52f7d0a952ed00418b3e372442efb21c7b2cd6b153e5ef
Sep 09 21:59:53.008380 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:53.008345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:53.047325 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:53.047290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z" event={"ID":"2e96ca93-b449-445e-b2a8-0dd4cb76ee78","Type":"ContainerStarted","Data":"1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158"}
Sep 09 21:59:53.047510 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:53.047343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z" event={"ID":"2e96ca93-b449-445e-b2a8-0dd4cb76ee78","Type":"ContainerStarted","Data":"5bf65be4a27a0778ce52f7d0a952ed00418b3e372442efb21c7b2cd6b153e5ef"}
Sep 09 21:59:53.047510 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:53.047405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 21:59:53.049061 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:53.049036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk" event={"ID":"71470ad2-0883-4473-8ab3-8c5e963871d1","Type":"ContainerStarted","Data":"76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47"}
Sep 09 21:59:53.049170 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:53.049110    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:53.072194 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:53.072142    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z" podStartSLOduration=2.072126383 podStartE2EDuration="2.072126383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:53.069587841 +0000 UTC m=+1285.966253848" watchObservedRunningTime="2026-09-09 21:59:53.072126383 +0000 UTC m=+1285.968792391"
Sep 09 21:59:53.094341 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:53.094294    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk" podStartSLOduration=2.6504736429999998 podStartE2EDuration="3.094278544s" totalImagesPullingTime="443.804901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:53.091298141 +0000 UTC m=+1285.987964149" watchObservedRunningTime="2026-09-09 21:59:53.094278544 +0000 UTC m=+1285.990944557"
Sep 09 21:59:59.060402 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:59.060372    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 21:59:59.891435 ip-10-0-129-45 kubenswrapper[2563]: I0909 21:59:59.891402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:00.600115 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.600066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:00.626757 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.626708    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"]
Sep 09 22:00:00.627131 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.626996    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9" podUID="3ef82d2b-d21b-41e5-841e-9a44a8b177a0" containerName="payload-processing" containerID="cri-o://b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab" gracePeriod=30
Sep 09 22:00:00.679075 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.679028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:00.777617 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.777581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:00.878419 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.878394    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:00.893486 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.893460    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 22:00:00.966954 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.966922    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-plugins-config-volume\") pod \"3ef82d2b-d21b-41e5-841e-9a44a8b177a0\" (UID: \"3ef82d2b-d21b-41e5-841e-9a44a8b177a0\") "
Sep 09 22:00:00.966954 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.966960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfnx2\" (UniqueName: \"kubernetes.io/projected/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-kube-api-access-cfnx2\") pod \"3ef82d2b-d21b-41e5-841e-9a44a8b177a0\" (UID: \"3ef82d2b-d21b-41e5-841e-9a44a8b177a0\") "
Sep 09 22:00:00.967386 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.967358    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ef82d2b-d21b-41e5-841e-9a44a8b177a0" (UID: "3ef82d2b-d21b-41e5-841e-9a44a8b177a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:00:00.969312 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:00.969288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-kube-api-access-cfnx2" (OuterVolumeSpecName: "kube-api-access-cfnx2") pod "3ef82d2b-d21b-41e5-841e-9a44a8b177a0" (UID: "3ef82d2b-d21b-41e5-841e-9a44a8b177a0"). InnerVolumeSpecName "kube-api-access-cfnx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:01.068207 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.068167    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:00:01.068207 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.068197    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfnx2\" (UniqueName: \"kubernetes.io/projected/3ef82d2b-d21b-41e5-841e-9a44a8b177a0-kube-api-access-cfnx2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:00:01.099089 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.099048    2563 generic.go:356] "Generic (PLEG): container finished" podID="3ef82d2b-d21b-41e5-841e-9a44a8b177a0" containerID="b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab" exitCode=0
Sep 09 22:00:01.099089 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.099092    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"
Sep 09 22:00:01.099266 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.099139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9" event={"ID":"3ef82d2b-d21b-41e5-841e-9a44a8b177a0","Type":"ContainerDied","Data":"b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab"}
Sep 09 22:00:01.099266 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.099192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9" event={"ID":"3ef82d2b-d21b-41e5-841e-9a44a8b177a0","Type":"ContainerDied","Data":"437f5d33d8470b25d01243b0911a268ad0dfcef5dadfe926d27d1629a92b2662"}
Sep 09 22:00:01.099266 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.099221    2563 scope.go:117] "RemoveContainer" containerID="b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab"
Sep 09 22:00:01.108840 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.108799    2563 scope.go:117] "RemoveContainer" containerID="b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab"
Sep 09 22:00:01.109143 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:00:01.109121    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab\": container with ID starting with b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab not found: ID does not exist" containerID="b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab"
Sep 09 22:00:01.109199 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.109154    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab"} err="failed to get container status \"b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab\": rpc error: code = NotFound desc = could not find container \"b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab\": container with ID starting with b742e4c35afe06d48ade9e87f6a8001766d83a4749b57ce218f8d4a2d386b5ab not found: ID does not exist"
Sep 09 22:00:01.122604 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.122524    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"]
Sep 09 22:00:01.127679 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.127652    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9"]
Sep 09 22:00:01.229865 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.229823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:01.606578 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.606537    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ef82d2b-d21b-41e5-841e-9a44a8b177a0" path="/var/lib/kubelet/pods/3ef82d2b-d21b-41e5-841e-9a44a8b177a0/volumes"
Sep 09 22:00:01.862857 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:01.862710    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-df8f95ff-58995595c-lqxb9" podUID="3ef82d2b-d21b-41e5-841e-9a44a8b177a0" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.69:9004: i/o timeout"
Sep 09 22:00:04.057591 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:04.057561    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 22:00:07.219691 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:07.219657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:07.295955 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:07.295922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:07.510684 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:07.510651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:07.862919 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:07.862827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:07.956538 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:07.956495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:08.072869 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:08.072834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:11.885406 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:11.885373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:14.009052 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:14.009013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:14.072279 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:14.072231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:14.153650 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:14.153607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:14.252885 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:14.252849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:14.352518 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:14.352414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:14.538277 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:14.538233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:18.359968 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.359926    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"]
Sep 09 22:00:18.362277 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.360390    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ef82d2b-d21b-41e5-841e-9a44a8b177a0" containerName="payload-processing"
Sep 09 22:00:18.362277 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.360402    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ef82d2b-d21b-41e5-841e-9a44a8b177a0" containerName="payload-processing"
Sep 09 22:00:18.362277 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.360481    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ef82d2b-d21b-41e5-841e-9a44a8b177a0" containerName="payload-processing"
Sep 09 22:00:18.363538 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.363517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.367522 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.367488    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ace629\"/\"openshift-service-ca.crt\""
Sep 09 22:00:18.367522 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.367506    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ace629\"/\"kube-root-ca.crt\""
Sep 09 22:00:18.367692 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.367488    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ace629\"/\"default-dockercfg-799s7\""
Sep 09 22:00:18.367692 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.367554    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ace629\"/\"auth-test-model-ace629-kserve-self-signed-certs\""
Sep 09 22:00:18.372200 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.372178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"]
Sep 09 22:00:18.429994 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.429960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-dshm\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.429994 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.430007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-home\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.430219 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.430139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tls-certs\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.430219 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.430192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tmp-dir\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.430309 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.430248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-model-cache\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.430355 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.430304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m44ll\" (UniqueName: \"kubernetes.io/projected/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-kube-api-access-m44ll\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.459950 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.459919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:18.531645 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.531604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tls-certs\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.531645 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.531649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tmp-dir\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.531939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.531672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-model-cache\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.531939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.531702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m44ll\" (UniqueName: \"kubernetes.io/projected/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-kube-api-access-m44ll\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.531939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.531739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-dshm\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.531939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.531806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-home\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.532155 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.532128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-model-cache\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.532206 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.532152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tmp-dir\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.532206 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.532196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-home\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.534143 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.534124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-dshm\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.534535 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.534512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tls-certs\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.540933 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.540909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m44ll\" (UniqueName: \"kubernetes.io/projected/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-kube-api-access-m44ll\") pod \"auth-test-model-ace629-kserve-645949d659-4gqz7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") " pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.674045 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.673943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:18.948359 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:18.948331    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"]
Sep 09 22:00:18.949422 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:00:18.949395    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6d7d803_7ce7_4f03_900e_fa96da4fe0b7.slice/crio-dd8c80cafa9c0a465a1d30834d3df1c0b12ab5b9cccd31012cbffa242dfa4177 WatchSource:0}: Error finding container dd8c80cafa9c0a465a1d30834d3df1c0b12ab5b9cccd31012cbffa242dfa4177: Status 404 returned error can't find the container with id dd8c80cafa9c0a465a1d30834d3df1c0b12ab5b9cccd31012cbffa242dfa4177
Sep 09 22:00:19.209743 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:19.209624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7" event={"ID":"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7","Type":"ContainerStarted","Data":"5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183"}
Sep 09 22:00:19.209743 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:19.209686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7" event={"ID":"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7","Type":"ContainerStarted","Data":"dd8c80cafa9c0a465a1d30834d3df1c0b12ab5b9cccd31012cbffa242dfa4177"}
Sep 09 22:00:19.228801 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:19.228731    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7" podStartSLOduration=1.228712577 podStartE2EDuration="1.228712577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:19.226745073 +0000 UTC m=+1312.123411082" watchObservedRunningTime="2026-09-09 22:00:19.228712577 +0000 UTC m=+1312.125378587"
Sep 09 22:00:20.191485 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:20.191456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:20.292339 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:20.292307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:21.122537 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:21.122506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:21.447161 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:21.447061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:21.568266 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:21.568235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:24.710177 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:24.710139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:24.771393 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:24.771358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:25.582322 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:25.582287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:25.685563 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:25.685532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:25.783051 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:25.783016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:26.145042 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:26.145009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:26.663325 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:26.663298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:28.674114 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:28.674078    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:28.674563 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:28.674136    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:28.686909 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:28.686875    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:29.279166 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:29.279131    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:00:44.054684 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.054644    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"]
Sep 09 22:00:44.059724 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.059704    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.063461 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.063434    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"openshift-service-ca.crt\""
Sep 09 22:00:44.063604 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.063440    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"kube-root-ca.crt\""
Sep 09 22:00:44.063604 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.063477    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"default-dockercfg-pkxqr\""
Sep 09 22:00:44.063604 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.063477    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"auth-test-model-3b2190-kserve-self-signed-certs\""
Sep 09 22:00:44.066429 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.066389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-model-cache\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.066557 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.066445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-tmp-dir\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.066557 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.066487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-dshm\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.066674 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.066565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdbx\" (UniqueName: \"kubernetes.io/projected/78a6f377-acde-4c50-a2e1-3526b51071e1-kube-api-access-zkdbx\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.066674 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.066613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a6f377-acde-4c50-a2e1-3526b51071e1-tls-certs\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.066674 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.066662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-home\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.069394 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.069365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"]
Sep 09 22:00:44.074266 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.074233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:44.167428 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.167389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-dshm\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.167622 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.167450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdbx\" (UniqueName: \"kubernetes.io/projected/78a6f377-acde-4c50-a2e1-3526b51071e1-kube-api-access-zkdbx\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.167622 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.167482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a6f377-acde-4c50-a2e1-3526b51071e1-tls-certs\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.167622 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.167510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-home\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.167622 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.167527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-model-cache\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.167622 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.167548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-tmp-dir\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.167958 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.167937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-home\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.168009 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.167974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-tmp-dir\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.168047 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.168008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-model-cache\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.169934 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.169915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-dshm\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.170212 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.170194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a6f377-acde-4c50-a2e1-3526b51071e1-tls-certs\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.176160 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.176138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdbx\" (UniqueName: \"kubernetes.io/projected/78a6f377-acde-4c50-a2e1-3526b51071e1-kube-api-access-zkdbx\") pod \"auth-test-model-3b2190-kserve-86f8f7c75f-f9dll\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.371315 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.371233    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:44.646180 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:44.646153    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"]
Sep 09 22:00:44.648219 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:00:44.648191    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78a6f377_acde_4c50_a2e1_3526b51071e1.slice/crio-ae25c034f29308ccd51016034c3153d22dd4749b75d169f01d190cdc60c534a0 WatchSource:0}: Error finding container ae25c034f29308ccd51016034c3153d22dd4749b75d169f01d190cdc60c534a0: Status 404 returned error can't find the container with id ae25c034f29308ccd51016034c3153d22dd4749b75d169f01d190cdc60c534a0
Sep 09 22:00:45.373164 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:45.373123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll" event={"ID":"78a6f377-acde-4c50-a2e1-3526b51071e1","Type":"ContainerStarted","Data":"7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b"}
Sep 09 22:00:45.373663 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:45.373172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll" event={"ID":"78a6f377-acde-4c50-a2e1-3526b51071e1","Type":"ContainerStarted","Data":"ae25c034f29308ccd51016034c3153d22dd4749b75d169f01d190cdc60c534a0"}
Sep 09 22:00:45.390508 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:45.390452    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll" podStartSLOduration=1.390436005 podStartE2EDuration="1.390436005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:45.389859311 +0000 UTC m=+1338.286525319" watchObservedRunningTime="2026-09-09 22:00:45.390436005 +0000 UTC m=+1338.287102013"
Sep 09 22:00:49.296021 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:49.295937    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:49.688419 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:49.688330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:50.003407 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:50.003368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:50.261076 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:50.260993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:53.474002 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:53.473960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:53.638037 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:53.638004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:54.371707 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:54.371668    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:54.371707 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:54.371715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:54.384562 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:54.384535    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:54.438742 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:54.438713    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:00:55.165266 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.165232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:55.262987 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.262936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:55.362886 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.362854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:55.414428 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.414390    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"]
Sep 09 22:00:55.418426 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.418379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:00:55.421190 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.421171    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-e5cb43\""
Sep 09 22:00:55.421301 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.421224    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-e5cb43-dockercfg-9zdmp\""
Sep 09 22:00:55.427105 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.427026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"]
Sep 09 22:00:55.470079 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.470047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmhh6\" (UniqueName: \"kubernetes.io/projected/cb3074a9-5070-4f5e-af1c-c44482d414ea-kube-api-access-gmhh6\") pod \"payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw\" (UID: \"cb3074a9-5070-4f5e-af1c-c44482d414ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:00:55.470307 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.470173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3074a9-5070-4f5e-af1c-c44482d414ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw\" (UID: \"cb3074a9-5070-4f5e-af1c-c44482d414ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:00:55.571154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.571119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3074a9-5070-4f5e-af1c-c44482d414ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw\" (UID: \"cb3074a9-5070-4f5e-af1c-c44482d414ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:00:55.571465 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.571200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmhh6\" (UniqueName: \"kubernetes.io/projected/cb3074a9-5070-4f5e-af1c-c44482d414ea-kube-api-access-gmhh6\") pod \"payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw\" (UID: \"cb3074a9-5070-4f5e-af1c-c44482d414ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:00:55.571901 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.571873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3074a9-5070-4f5e-af1c-c44482d414ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw\" (UID: \"cb3074a9-5070-4f5e-af1c-c44482d414ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:00:55.580222 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.580196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmhh6\" (UniqueName: \"kubernetes.io/projected/cb3074a9-5070-4f5e-af1c-c44482d414ea-kube-api-access-gmhh6\") pod \"payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw\" (UID: \"cb3074a9-5070-4f5e-af1c-c44482d414ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:00:55.732388 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:55.732288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:00:56.012168 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:56.012135    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"]
Sep 09 22:00:56.013664 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:00:56.013638    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3074a9_5070_4f5e_af1c_c44482d414ea.slice/crio-896e436e6bdb390a42b0ed9de79adab56d7c766c846b387c438107f440eca91a WatchSource:0}: Error finding container 896e436e6bdb390a42b0ed9de79adab56d7c766c846b387c438107f440eca91a: Status 404 returned error can't find the container with id 896e436e6bdb390a42b0ed9de79adab56d7c766c846b387c438107f440eca91a
Sep 09 22:00:56.444026 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:56.443920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw" event={"ID":"cb3074a9-5070-4f5e-af1c-c44482d414ea","Type":"ContainerStarted","Data":"61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e"}
Sep 09 22:00:56.444026 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:56.443977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw" event={"ID":"cb3074a9-5070-4f5e-af1c-c44482d414ea","Type":"ContainerStarted","Data":"896e436e6bdb390a42b0ed9de79adab56d7c766c846b387c438107f440eca91a"}
Sep 09 22:00:56.444435 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:56.444035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:00:56.462519 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:00:56.462459    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw" podStartSLOduration=1.462441879 podStartE2EDuration="1.462441879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:56.460764708 +0000 UTC m=+1349.357430723" watchObservedRunningTime="2026-09-09 22:00:56.462441879 +0000 UTC m=+1349.359107887"
Sep 09 22:01:06.952146 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:06.952114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:07.052375 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:07.052338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:07.452540 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:07.452509    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:01:11.249597 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:11.249553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:11.351545 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:11.351512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:18.439747 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:18.439710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:19.057458 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:19.057418    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-139977364690528"]
Sep 09 22:01:19.061495 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:19.061464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:19.064392 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:19.064366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fkr9w\""
Sep 09 22:01:19.065567 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:19.065542    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-139977364690528"]
Sep 09 22:01:19.208288 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:19.208253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rz28\" (UniqueName: \"kubernetes.io/projected/bd203dcd-43d7-4a0f-8445-4a6633c21e98-kube-api-access-2rz28\") pod \"test-curl-7761-139977364690528\" (UID: \"bd203dcd-43d7-4a0f-8445-4a6633c21e98\") " pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:19.308925 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:19.308833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rz28\" (UniqueName: \"kubernetes.io/projected/bd203dcd-43d7-4a0f-8445-4a6633c21e98-kube-api-access-2rz28\") pod \"test-curl-7761-139977364690528\" (UID: \"bd203dcd-43d7-4a0f-8445-4a6633c21e98\") " pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:19.317544 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:19.317511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rz28\" (UniqueName: \"kubernetes.io/projected/bd203dcd-43d7-4a0f-8445-4a6633c21e98-kube-api-access-2rz28\") pod \"test-curl-7761-139977364690528\" (UID: \"bd203dcd-43d7-4a0f-8445-4a6633c21e98\") " pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:19.371318 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:19.371273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:19.642605 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:19.642576    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-139977364690528"]
Sep 09 22:01:19.643912 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:01:19.643883    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd203dcd_43d7_4a0f_8445_4a6633c21e98.slice/crio-f04cf7fba6d42edb4bbe2278c93fd6a069f2abce0a463a7febc07d71b21b7bad WatchSource:0}: Error finding container f04cf7fba6d42edb4bbe2278c93fd6a069f2abce0a463a7febc07d71b21b7bad: Status 404 returned error can't find the container with id f04cf7fba6d42edb4bbe2278c93fd6a069f2abce0a463a7febc07d71b21b7bad
Sep 09 22:01:20.598321 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:20.598212    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd203dcd-43d7-4a0f-8445-4a6633c21e98" containerID="ce767bfbb8b7a328ee83a230068e09dfc91d626a4cc1cd9f7aaf41c09a0b8787" exitCode=0
Sep 09 22:01:20.598321 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:20.598299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-139977364690528" event={"ID":"bd203dcd-43d7-4a0f-8445-4a6633c21e98","Type":"ContainerDied","Data":"ce767bfbb8b7a328ee83a230068e09dfc91d626a4cc1cd9f7aaf41c09a0b8787"}
Sep 09 22:01:20.598538 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:20.598352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-139977364690528" event={"ID":"bd203dcd-43d7-4a0f-8445-4a6633c21e98","Type":"ContainerStarted","Data":"f04cf7fba6d42edb4bbe2278c93fd6a069f2abce0a463a7febc07d71b21b7bad"}
Sep 09 22:01:21.743407 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.743380    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:21.762643 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.762616    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7761-139977364690528_bd203dcd-43d7-4a0f-8445-4a6633c21e98/test-curl-7761-139977364690528/0.log"
Sep 09 22:01:21.785349 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.785313    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-139977364690528"]
Sep 09 22:01:21.787830 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.787803    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-139977364690528"]
Sep 09 22:01:21.832027 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.831997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rz28\" (UniqueName: \"kubernetes.io/projected/bd203dcd-43d7-4a0f-8445-4a6633c21e98-kube-api-access-2rz28\") pod \"bd203dcd-43d7-4a0f-8445-4a6633c21e98\" (UID: \"bd203dcd-43d7-4a0f-8445-4a6633c21e98\") "
Sep 09 22:01:21.834167 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.834141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd203dcd-43d7-4a0f-8445-4a6633c21e98-kube-api-access-2rz28" (OuterVolumeSpecName: "kube-api-access-2rz28") pod "bd203dcd-43d7-4a0f-8445-4a6633c21e98" (UID: "bd203dcd-43d7-4a0f-8445-4a6633c21e98"). InnerVolumeSpecName "kube-api-access-2rz28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:01:21.892224 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.892133    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-139977364690528"]
Sep 09 22:01:21.892648 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.892628    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd203dcd-43d7-4a0f-8445-4a6633c21e98" containerName="test-curl-7761-139977364690528"
Sep 09 22:01:21.892736 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.892652    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd203dcd-43d7-4a0f-8445-4a6633c21e98" containerName="test-curl-7761-139977364690528"
Sep 09 22:01:21.892832 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.892817    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd203dcd-43d7-4a0f-8445-4a6633c21e98" containerName="test-curl-7761-139977364690528"
Sep 09 22:01:21.896006 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.895987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:21.900477 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.900448    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-139977364690528"]
Sep 09 22:01:21.932725 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.932690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf27k\" (UniqueName: \"kubernetes.io/projected/70f750bb-b456-4797-a94b-086b3c43b74a-kube-api-access-qf27k\") pod \"test-curl-7761-139977364690528\" (UID: \"70f750bb-b456-4797-a94b-086b3c43b74a\") " pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:21.932915 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:21.932799    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rz28\" (UniqueName: \"kubernetes.io/projected/bd203dcd-43d7-4a0f-8445-4a6633c21e98-kube-api-access-2rz28\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:01:22.033669 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:22.033633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf27k\" (UniqueName: \"kubernetes.io/projected/70f750bb-b456-4797-a94b-086b3c43b74a-kube-api-access-qf27k\") pod \"test-curl-7761-139977364690528\" (UID: \"70f750bb-b456-4797-a94b-086b3c43b74a\") " pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:22.043500 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:22.043472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf27k\" (UniqueName: \"kubernetes.io/projected/70f750bb-b456-4797-a94b-086b3c43b74a-kube-api-access-qf27k\") pod \"test-curl-7761-139977364690528\" (UID: \"70f750bb-b456-4797-a94b-086b3c43b74a\") " pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:22.208172 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:22.208082    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:22.480344 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:22.480307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-139977364690528"]
Sep 09 22:01:22.483949 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:01:22.482794    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70f750bb_b456_4797_a94b_086b3c43b74a.slice/crio-7b46f7dc0d0d23353b6edf168794d72a5075374a7f28fd7004869a8e123b238e WatchSource:0}: Error finding container 7b46f7dc0d0d23353b6edf168794d72a5075374a7f28fd7004869a8e123b238e: Status 404 returned error can't find the container with id 7b46f7dc0d0d23353b6edf168794d72a5075374a7f28fd7004869a8e123b238e
Sep 09 22:01:22.610600 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:22.610560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-139977364690528" event={"ID":"70f750bb-b456-4797-a94b-086b3c43b74a","Type":"ContainerStarted","Data":"7b46f7dc0d0d23353b6edf168794d72a5075374a7f28fd7004869a8e123b238e"}
Sep 09 22:01:22.611930 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:22.611908    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f04cf7fba6d42edb4bbe2278c93fd6a069f2abce0a463a7febc07d71b21b7bad"
Sep 09 22:01:22.611930 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:22.611910    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:22.623712 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:22.623685    2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7761-139977364690528" oldPodUID="bd203dcd-43d7-4a0f-8445-4a6633c21e98" podUID="70f750bb-b456-4797-a94b-086b3c43b74a"
Sep 09 22:01:23.607124 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:23.607044    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd203dcd-43d7-4a0f-8445-4a6633c21e98" path="/var/lib/kubelet/pods/bd203dcd-43d7-4a0f-8445-4a6633c21e98/volumes"
Sep 09 22:01:23.618897 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:23.618872    2563 generic.go:356] "Generic (PLEG): container finished" podID="70f750bb-b456-4797-a94b-086b3c43b74a" containerID="75dd9b6065242f277085dfa7da89fa7d8b3c1932e6c27b3cf39d6a56f1d9d165" exitCode=0
Sep 09 22:01:23.619021 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:23.618966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-139977364690528" event={"ID":"70f750bb-b456-4797-a94b-086b3c43b74a","Type":"ContainerDied","Data":"75dd9b6065242f277085dfa7da89fa7d8b3c1932e6c27b3cf39d6a56f1d9d165"}
Sep 09 22:01:24.761646 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.761609    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:24.782065 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.782038    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7761-139977364690528_70f750bb-b456-4797-a94b-086b3c43b74a/test-curl-7761-139977364690528/0.log"
Sep 09 22:01:24.805266 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.805233    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-139977364690528"]
Sep 09 22:01:24.808936 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.808907    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-139977364690528"]
Sep 09 22:01:24.858115 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.858080    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf27k\" (UniqueName: \"kubernetes.io/projected/70f750bb-b456-4797-a94b-086b3c43b74a-kube-api-access-qf27k\") pod \"70f750bb-b456-4797-a94b-086b3c43b74a\" (UID: \"70f750bb-b456-4797-a94b-086b3c43b74a\") "
Sep 09 22:01:24.860251 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.860227    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f750bb-b456-4797-a94b-086b3c43b74a-kube-api-access-qf27k" (OuterVolumeSpecName: "kube-api-access-qf27k") pod "70f750bb-b456-4797-a94b-086b3c43b74a" (UID: "70f750bb-b456-4797-a94b-086b3c43b74a"). InnerVolumeSpecName "kube-api-access-qf27k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:01:24.926298 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.926206    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-139977364691104"]
Sep 09 22:01:24.926718 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.926701    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f750bb-b456-4797-a94b-086b3c43b74a" containerName="test-curl-7761-139977364690528"
Sep 09 22:01:24.926838 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.926721    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f750bb-b456-4797-a94b-086b3c43b74a" containerName="test-curl-7761-139977364690528"
Sep 09 22:01:24.926912 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.926849    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f750bb-b456-4797-a94b-086b3c43b74a" containerName="test-curl-7761-139977364690528"
Sep 09 22:01:24.931051 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.931022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:24.939652 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.939618    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-139977364691104"]
Sep 09 22:01:24.959402 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.959370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2wv\" (UniqueName: \"kubernetes.io/projected/c2ab76ee-fdbd-4302-91a1-85d61ff14f46-kube-api-access-bw2wv\") pod \"test-curl-7761-139977364691104\" (UID: \"c2ab76ee-fdbd-4302-91a1-85d61ff14f46\") " pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:24.959542 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:24.959431    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf27k\" (UniqueName: \"kubernetes.io/projected/70f750bb-b456-4797-a94b-086b3c43b74a-kube-api-access-qf27k\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:01:25.060751 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:25.060711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2wv\" (UniqueName: \"kubernetes.io/projected/c2ab76ee-fdbd-4302-91a1-85d61ff14f46-kube-api-access-bw2wv\") pod \"test-curl-7761-139977364691104\" (UID: \"c2ab76ee-fdbd-4302-91a1-85d61ff14f46\") " pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:25.069130 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:25.069101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2wv\" (UniqueName: \"kubernetes.io/projected/c2ab76ee-fdbd-4302-91a1-85d61ff14f46-kube-api-access-bw2wv\") pod \"test-curl-7761-139977364691104\" (UID: \"c2ab76ee-fdbd-4302-91a1-85d61ff14f46\") " pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:25.245511 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:25.245483    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:25.517939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:25.517907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-139977364691104"]
Sep 09 22:01:25.519084 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:01:25.519050    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ab76ee_fdbd_4302_91a1_85d61ff14f46.slice/crio-25b114bb24d558f165e43b84686c3fc681a45326aa967869b6037b6397a318e4 WatchSource:0}: Error finding container 25b114bb24d558f165e43b84686c3fc681a45326aa967869b6037b6397a318e4: Status 404 returned error can't find the container with id 25b114bb24d558f165e43b84686c3fc681a45326aa967869b6037b6397a318e4
Sep 09 22:01:25.606621 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:25.606591    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f750bb-b456-4797-a94b-086b3c43b74a" path="/var/lib/kubelet/pods/70f750bb-b456-4797-a94b-086b3c43b74a/volumes"
Sep 09 22:01:25.631791 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:25.631741    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364690528"
Sep 09 22:01:25.631945 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:25.631799    2563 scope.go:117] "RemoveContainer" containerID="75dd9b6065242f277085dfa7da89fa7d8b3c1932e6c27b3cf39d6a56f1d9d165"
Sep 09 22:01:25.633157 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:25.633135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-139977364691104" event={"ID":"c2ab76ee-fdbd-4302-91a1-85d61ff14f46","Type":"ContainerStarted","Data":"25b114bb24d558f165e43b84686c3fc681a45326aa967869b6037b6397a318e4"}
Sep 09 22:01:26.642133 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:26.642100    2563 generic.go:356] "Generic (PLEG): container finished" podID="c2ab76ee-fdbd-4302-91a1-85d61ff14f46" containerID="ddaad823dc84a9eca17edec2a5fe357da7a36ca1b9896071fa5eb20215018221" exitCode=0
Sep 09 22:01:26.642533 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:26.642140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-139977364691104" event={"ID":"c2ab76ee-fdbd-4302-91a1-85d61ff14f46","Type":"ContainerDied","Data":"ddaad823dc84a9eca17edec2a5fe357da7a36ca1b9896071fa5eb20215018221"}
Sep 09 22:01:27.737719 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.737689    2563 scope.go:117] "RemoveContainer" containerID="411477aa62867bcc4aa415a330d37659125b96f76f8ce31945d89f03951ffb12"
Sep 09 22:01:27.792347 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.792323    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:27.813074 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.813042    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7761-139977364691104_c2ab76ee-fdbd-4302-91a1-85d61ff14f46/test-curl-7761-139977364691104/0.log"
Sep 09 22:01:27.838275 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.838242    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-139977364691104"]
Sep 09 22:01:27.842307 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.842281    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-139977364691104"]
Sep 09 22:01:27.885591 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.885560    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw2wv\" (UniqueName: \"kubernetes.io/projected/c2ab76ee-fdbd-4302-91a1-85d61ff14f46-kube-api-access-bw2wv\") pod \"c2ab76ee-fdbd-4302-91a1-85d61ff14f46\" (UID: \"c2ab76ee-fdbd-4302-91a1-85d61ff14f46\") "
Sep 09 22:01:27.887777 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.887739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ab76ee-fdbd-4302-91a1-85d61ff14f46-kube-api-access-bw2wv" (OuterVolumeSpecName: "kube-api-access-bw2wv") pod "c2ab76ee-fdbd-4302-91a1-85d61ff14f46" (UID: "c2ab76ee-fdbd-4302-91a1-85d61ff14f46"). InnerVolumeSpecName "kube-api-access-bw2wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:01:27.954375 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.954302    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-139977364691104"]
Sep 09 22:01:27.954712 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.954699    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ab76ee-fdbd-4302-91a1-85d61ff14f46" containerName="test-curl-7761-139977364691104"
Sep 09 22:01:27.954755 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.954713    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ab76ee-fdbd-4302-91a1-85d61ff14f46" containerName="test-curl-7761-139977364691104"
Sep 09 22:01:27.954826 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.954817    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ab76ee-fdbd-4302-91a1-85d61ff14f46" containerName="test-curl-7761-139977364691104"
Sep 09 22:01:27.958058 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.958038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:27.961623 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.961590    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-139977364691104"]
Sep 09 22:01:27.986568 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.986539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9rfz\" (UniqueName: \"kubernetes.io/projected/e426734b-25a8-4d7e-8329-db24b5e6fdf6-kube-api-access-k9rfz\") pod \"test-curl-7761-139977364691104\" (UID: \"e426734b-25a8-4d7e-8329-db24b5e6fdf6\") " pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:27.986704 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:27.986596    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw2wv\" (UniqueName: \"kubernetes.io/projected/c2ab76ee-fdbd-4302-91a1-85d61ff14f46-kube-api-access-bw2wv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:01:28.087159 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:28.087106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9rfz\" (UniqueName: \"kubernetes.io/projected/e426734b-25a8-4d7e-8329-db24b5e6fdf6-kube-api-access-k9rfz\") pod \"test-curl-7761-139977364691104\" (UID: \"e426734b-25a8-4d7e-8329-db24b5e6fdf6\") " pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:28.096915 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:28.096884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9rfz\" (UniqueName: \"kubernetes.io/projected/e426734b-25a8-4d7e-8329-db24b5e6fdf6-kube-api-access-k9rfz\") pod \"test-curl-7761-139977364691104\" (UID: \"e426734b-25a8-4d7e-8329-db24b5e6fdf6\") " pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:28.268610 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:28.268570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:28.540426 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:28.540399    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-139977364691104"]
Sep 09 22:01:28.541476 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:01:28.541450    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode426734b_25a8_4d7e_8329_db24b5e6fdf6.slice/crio-c77ddc4c2c1e47fc827a2044694adbe9ad3a945055e139bacae179e055899b19 WatchSource:0}: Error finding container c77ddc4c2c1e47fc827a2044694adbe9ad3a945055e139bacae179e055899b19: Status 404 returned error can't find the container with id c77ddc4c2c1e47fc827a2044694adbe9ad3a945055e139bacae179e055899b19
Sep 09 22:01:28.655764 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:28.655730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-139977364691104" event={"ID":"e426734b-25a8-4d7e-8329-db24b5e6fdf6","Type":"ContainerStarted","Data":"c77ddc4c2c1e47fc827a2044694adbe9ad3a945055e139bacae179e055899b19"}
Sep 09 22:01:28.657123 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:28.657103    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:28.657123 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:28.657119    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25b114bb24d558f165e43b84686c3fc681a45326aa967869b6037b6397a318e4"
Sep 09 22:01:28.668605 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:28.668581    2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7761-139977364691104" oldPodUID="c2ab76ee-fdbd-4302-91a1-85d61ff14f46" podUID="e426734b-25a8-4d7e-8329-db24b5e6fdf6"
Sep 09 22:01:29.606922 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:29.606888    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ab76ee-fdbd-4302-91a1-85d61ff14f46" path="/var/lib/kubelet/pods/c2ab76ee-fdbd-4302-91a1-85d61ff14f46/volumes"
Sep 09 22:01:29.664330 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:29.664292    2563 generic.go:356] "Generic (PLEG): container finished" podID="e426734b-25a8-4d7e-8329-db24b5e6fdf6" containerID="391f1f1ef4d7001e1367f3bdb1bf1d7fedb3fe9c9cceea937981397e676b4324" exitCode=0
Sep 09 22:01:29.664498 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:29.664356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-139977364691104" event={"ID":"e426734b-25a8-4d7e-8329-db24b5e6fdf6","Type":"ContainerDied","Data":"391f1f1ef4d7001e1367f3bdb1bf1d7fedb3fe9c9cceea937981397e676b4324"}
Sep 09 22:01:30.808701 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:30.808672    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:30.828571 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:30.828532    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7761-139977364691104_e426734b-25a8-4d7e-8329-db24b5e6fdf6/test-curl-7761-139977364691104/0.log"
Sep 09 22:01:30.851121 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:30.851090    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-139977364691104"]
Sep 09 22:01:30.854928 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:30.854903    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-139977364691104"]
Sep 09 22:01:30.913685 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:30.913652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9rfz\" (UniqueName: \"kubernetes.io/projected/e426734b-25a8-4d7e-8329-db24b5e6fdf6-kube-api-access-k9rfz\") pod \"e426734b-25a8-4d7e-8329-db24b5e6fdf6\" (UID: \"e426734b-25a8-4d7e-8329-db24b5e6fdf6\") "
Sep 09 22:01:30.915909 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:30.915882    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e426734b-25a8-4d7e-8329-db24b5e6fdf6-kube-api-access-k9rfz" (OuterVolumeSpecName: "kube-api-access-k9rfz") pod "e426734b-25a8-4d7e-8329-db24b5e6fdf6" (UID: "e426734b-25a8-4d7e-8329-db24b5e6fdf6"). InnerVolumeSpecName "kube-api-access-k9rfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:01:31.014909 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:31.014874    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9rfz\" (UniqueName: \"kubernetes.io/projected/e426734b-25a8-4d7e-8329-db24b5e6fdf6-kube-api-access-k9rfz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:01:31.428994 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:31.428957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:31.606195 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:31.606160    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e426734b-25a8-4d7e-8329-db24b5e6fdf6" path="/var/lib/kubelet/pods/e426734b-25a8-4d7e-8329-db24b5e6fdf6/volumes"
Sep 09 22:01:31.676634 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:31.676611    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-139977364691104"
Sep 09 22:01:31.676835 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:31.676664    2563 scope.go:117] "RemoveContainer" containerID="391f1f1ef4d7001e1367f3bdb1bf1d7fedb3fe9c9cceea937981397e676b4324"
Sep 09 22:01:31.741594 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:31.741561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:31.843487 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:31.843460    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:34.847684 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:34.847647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:34.940207 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:34.940175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:36.171318 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.171278    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"]
Sep 09 22:01:36.172257 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.172218    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e426734b-25a8-4d7e-8329-db24b5e6fdf6" containerName="test-curl-7761-139977364691104"
Sep 09 22:01:36.172420 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.172404    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e426734b-25a8-4d7e-8329-db24b5e6fdf6" containerName="test-curl-7761-139977364691104"
Sep 09 22:01:36.172837 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.172813    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e426734b-25a8-4d7e-8329-db24b5e6fdf6" containerName="test-curl-7761-139977364691104"
Sep 09 22:01:36.180831 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.180808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.185392 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.185352    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6a897d5c-serving-cert\""
Sep 09 22:01:36.187434 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.187402    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"]
Sep 09 22:01:36.209039 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.209005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:36.266059 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.266017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d33b5b9-4898-4f84-8850-b4d0915282e6-openshift-service-ca\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.266280 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.266133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cqj\" (UniqueName: \"kubernetes.io/projected/3d33b5b9-4898-4f84-8850-b4d0915282e6-kube-api-access-v9cqj\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.266280 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.266179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-metrics-certs\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.266280 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.266213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-maas-api-tls\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.318203 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.318167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:36.366723 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.366689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d33b5b9-4898-4f84-8850-b4d0915282e6-openshift-service-ca\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.366947 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.366819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cqj\" (UniqueName: \"kubernetes.io/projected/3d33b5b9-4898-4f84-8850-b4d0915282e6-kube-api-access-v9cqj\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.366947 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.366893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-metrics-certs\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.367089 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.367061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-maas-api-tls\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.367472 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.367451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d33b5b9-4898-4f84-8850-b4d0915282e6-openshift-service-ca\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.369420 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.369399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-metrics-certs\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.369526 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.369510    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-maas-api-tls\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.375851 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.375829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cqj\" (UniqueName: \"kubernetes.io/projected/3d33b5b9-4898-4f84-8850-b4d0915282e6-kube-api-access-v9cqj\") pod \"maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.415669 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.415638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:36.495426 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.495395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:36.723849 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.723812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:36.777059 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.777028    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"]
Sep 09 22:01:36.778540 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:01:36.778516    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d33b5b9_4898_4f84_8850_b4d0915282e6.slice/crio-8f93d23b41f6f0508a731afe4353c9c151d993dcbe1936b42cfa6c6da6b534d0 WatchSource:0}: Error finding container 8f93d23b41f6f0508a731afe4353c9c151d993dcbe1936b42cfa6c6da6b534d0: Status 404 returned error can't find the container with id 8f93d23b41f6f0508a731afe4353c9c151d993dcbe1936b42cfa6c6da6b534d0
Sep 09 22:01:36.823685 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:36.823652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:37.716357 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:37.716272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn" event={"ID":"3d33b5b9-4898-4f84-8850-b4d0915282e6","Type":"ContainerStarted","Data":"9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c"}
Sep 09 22:01:37.716357 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:37.716320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn" event={"ID":"3d33b5b9-4898-4f84-8850-b4d0915282e6","Type":"ContainerStarted","Data":"8f93d23b41f6f0508a731afe4353c9c151d993dcbe1936b42cfa6c6da6b534d0"}
Sep 09 22:01:37.716357 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:37.716352    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:37.740524 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:37.740472    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn" podStartSLOduration=1.211850955 podStartE2EDuration="1.740455465s" totalImagesPullingTime="528.60451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:01:37.738743941 +0000 UTC m=+1390.635409950" watchObservedRunningTime="2026-09-09 22:01:37.740455465 +0000 UTC m=+1390.637121522"
Sep 09 22:01:38.220527 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:38.220494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:43.728653 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:43.728621    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:01:59.665666 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:01:59.665631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:00.783557 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:00.783525    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:17.765099 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.765063    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"]
Sep 09 22:02:17.769174 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.769156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.773280 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.773248    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6a897d5c\"/\"kube-root-ca.crt\""
Sep 09 22:02:17.773428 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.773248    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6a897d5c\"/\"e2e-lifecycle-model-6a897d5c-kserve-self-signed-certs\""
Sep 09 22:02:17.773428 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.773249    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6a897d5c\"/\"default-dockercfg-8mlh4\""
Sep 09 22:02:17.773566 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.773545    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6a897d5c\"/\"openshift-service-ca.crt\""
Sep 09 22:02:17.782343 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.782321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"]
Sep 09 22:02:17.882228 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.882188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-home\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.882228 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.882226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-dshm\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.882428 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.882250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a27e86-277c-4b11-8527-2a757d476d5a-tls-certs\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.882428 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.882347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-model-cache\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.882428 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.882388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-tmp-dir\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.882532 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.882473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgtq\" (UniqueName: \"kubernetes.io/projected/58a27e86-277c-4b11-8527-2a757d476d5a-kube-api-access-2sgtq\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.983861 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.983808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-home\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.983861 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.983870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-dshm\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.984115 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.983899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a27e86-277c-4b11-8527-2a757d476d5a-tls-certs\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.984115 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.984051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-model-cache\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.984115 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.984105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-tmp-dir\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.984248 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.984179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgtq\" (UniqueName: \"kubernetes.io/projected/58a27e86-277c-4b11-8527-2a757d476d5a-kube-api-access-2sgtq\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.984300 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.984266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-home\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.984448 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.984426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-model-cache\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.984515 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.984459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-tmp-dir\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.986378 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.986352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-dshm\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.986649 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.986629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a27e86-277c-4b11-8527-2a757d476d5a-tls-certs\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:17.993635 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:17.993615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgtq\" (UniqueName: \"kubernetes.io/projected/58a27e86-277c-4b11-8527-2a757d476d5a-kube-api-access-2sgtq\") pod \"e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") " pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:18.079501 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:18.079411    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:18.362437 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:18.362409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"]
Sep 09 22:02:18.363433 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:02:18.363407    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58a27e86_277c_4b11_8527_2a757d476d5a.slice/crio-b5199a9b519d0f5cca155f1ded28732564c25c7b4d0105eec6eaa4ab03ad4f05 WatchSource:0}: Error finding container b5199a9b519d0f5cca155f1ded28732564c25c7b4d0105eec6eaa4ab03ad4f05: Status 404 returned error can't find the container with id b5199a9b519d0f5cca155f1ded28732564c25c7b4d0105eec6eaa4ab03ad4f05
Sep 09 22:02:18.910636 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:18.910596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:18.960356 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:18.960316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96" event={"ID":"58a27e86-277c-4b11-8527-2a757d476d5a","Type":"ContainerStarted","Data":"8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864"}
Sep 09 22:02:18.960553 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:18.960369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96" event={"ID":"58a27e86-277c-4b11-8527-2a757d476d5a","Type":"ContainerStarted","Data":"b5199a9b519d0f5cca155f1ded28732564c25c7b4d0105eec6eaa4ab03ad4f05"}
Sep 09 22:02:18.989464 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:18.989412    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96" podStartSLOduration=1.98939706 podStartE2EDuration="1.98939706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:02:18.987226254 +0000 UTC m=+1431.883892264" watchObservedRunningTime="2026-09-09 22:02:18.98939706 +0000 UTC m=+1431.886063087"
Sep 09 22:02:28.079598 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:28.079514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:28.079598 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:28.079596    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:28.092558 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:28.092532    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:29.033392 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:29.033355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:02:39.095451 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:39.095420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:41.649217 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:41.649182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:41.716430 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:41.716396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:54.430997 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.430957    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"]
Sep 09 22:02:54.431499 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.431213    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn" podUID="3d33b5b9-4898-4f84-8850-b4d0915282e6" containerName="maas-api" containerID="cri-o://9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c" gracePeriod=30
Sep 09 22:02:54.461050 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.461021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:54.590554 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.590516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:54.689302 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.689238    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:02:54.727097 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.727062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:54.733201 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.733178    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-maas-api-tls\") pod \"3d33b5b9-4898-4f84-8850-b4d0915282e6\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") "
Sep 09 22:02:54.733372 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.733212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d33b5b9-4898-4f84-8850-b4d0915282e6-openshift-service-ca\") pod \"3d33b5b9-4898-4f84-8850-b4d0915282e6\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") "
Sep 09 22:02:54.733372 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.733259    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9cqj\" (UniqueName: \"kubernetes.io/projected/3d33b5b9-4898-4f84-8850-b4d0915282e6-kube-api-access-v9cqj\") pod \"3d33b5b9-4898-4f84-8850-b4d0915282e6\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") "
Sep 09 22:02:54.733372 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.733359    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-metrics-certs\") pod \"3d33b5b9-4898-4f84-8850-b4d0915282e6\" (UID: \"3d33b5b9-4898-4f84-8850-b4d0915282e6\") "
Sep 09 22:02:54.733731 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.733637    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d33b5b9-4898-4f84-8850-b4d0915282e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d33b5b9-4898-4f84-8850-b4d0915282e6" (UID: "3d33b5b9-4898-4f84-8850-b4d0915282e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:02:54.733873 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.733739    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d33b5b9-4898-4f84-8850-b4d0915282e6-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:02:54.735695 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.735664    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d33b5b9-4898-4f84-8850-b4d0915282e6" (UID: "3d33b5b9-4898-4f84-8850-b4d0915282e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:02:54.735848 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.735787    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3d33b5b9-4898-4f84-8850-b4d0915282e6" (UID: "3d33b5b9-4898-4f84-8850-b4d0915282e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:02:54.735848 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.735801    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d33b5b9-4898-4f84-8850-b4d0915282e6-kube-api-access-v9cqj" (OuterVolumeSpecName: "kube-api-access-v9cqj") pod "3d33b5b9-4898-4f84-8850-b4d0915282e6" (UID: "3d33b5b9-4898-4f84-8850-b4d0915282e6"). InnerVolumeSpecName "kube-api-access-v9cqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:02:54.808237 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.808199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:54.834610 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.834572    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9cqj\" (UniqueName: \"kubernetes.io/projected/3d33b5b9-4898-4f84-8850-b4d0915282e6-kube-api-access-v9cqj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:02:54.834610 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.834603    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:02:54.834610 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:54.834616    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d33b5b9-4898-4f84-8850-b4d0915282e6-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:02:55.177648 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:55.177614    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d33b5b9-4898-4f84-8850-b4d0915282e6" containerID="9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c" exitCode=0
Sep 09 22:02:55.177854 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:55.177660    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"
Sep 09 22:02:55.177854 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:55.177670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn" event={"ID":"3d33b5b9-4898-4f84-8850-b4d0915282e6","Type":"ContainerDied","Data":"9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c"}
Sep 09 22:02:55.177854 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:55.177713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn" event={"ID":"3d33b5b9-4898-4f84-8850-b4d0915282e6","Type":"ContainerDied","Data":"8f93d23b41f6f0508a731afe4353c9c151d993dcbe1936b42cfa6c6da6b534d0"}
Sep 09 22:02:55.177854 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:55.177735    2563 scope.go:117] "RemoveContainer" containerID="9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c"
Sep 09 22:02:55.187938 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:55.187920    2563 scope.go:117] "RemoveContainer" containerID="9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c"
Sep 09 22:02:55.188173 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:02:55.188152    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c\": container with ID starting with 9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c not found: ID does not exist" containerID="9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c"
Sep 09 22:02:55.188231 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:55.188181    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c"} err="failed to get container status \"9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c\": rpc error: code = NotFound desc = could not find container \"9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c\": container with ID starting with 9929c8c12925a02c464c028b8910c4508677ff23a93907601368c499128dbc2c not found: ID does not exist"
Sep 09 22:02:55.199759 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:55.199736    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"]
Sep 09 22:02:55.203474 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:55.203450    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a897d5c-7d9c594965-hsrdn"]
Sep 09 22:02:55.611707 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:55.611668    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d33b5b9-4898-4f84-8850-b4d0915282e6" path="/var/lib/kubelet/pods/3d33b5b9-4898-4f84-8850-b4d0915282e6/volumes"
Sep 09 22:02:57.929811 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:02:57.929759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:02.911307 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:02.911270    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"]
Sep 09 22:03:03.005024 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:03.004988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:13.823786 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:13.823739    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7764-93f252"]
Sep 09 22:03:13.824212 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:13.824198    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d33b5b9-4898-4f84-8850-b4d0915282e6" containerName="maas-api"
Sep 09 22:03:13.824261 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:13.824214    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d33b5b9-4898-4f84-8850-b4d0915282e6" containerName="maas-api"
Sep 09 22:03:13.824296 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:13.824286    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d33b5b9-4898-4f84-8850-b4d0915282e6" containerName="maas-api"
Sep 09 22:03:13.827559 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:13.827529    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-93f252"
Sep 09 22:03:13.830352 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:13.830325    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fkr9w\""
Sep 09 22:03:13.831642 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:13.831620    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7764-93f252"]
Sep 09 22:03:13.921379 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:13.921347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9vmm\" (UniqueName: \"kubernetes.io/projected/65537e68-a868-41a3-99a5-d6b7ec61ec9a-kube-api-access-d9vmm\") pod \"mt-curl-7764-93f252\" (UID: \"65537e68-a868-41a3-99a5-d6b7ec61ec9a\") " pod="opendatahub/mt-curl-7764-93f252"
Sep 09 22:03:14.022155 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:14.022119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9vmm\" (UniqueName: \"kubernetes.io/projected/65537e68-a868-41a3-99a5-d6b7ec61ec9a-kube-api-access-d9vmm\") pod \"mt-curl-7764-93f252\" (UID: \"65537e68-a868-41a3-99a5-d6b7ec61ec9a\") " pod="opendatahub/mt-curl-7764-93f252"
Sep 09 22:03:14.030671 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:14.030637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9vmm\" (UniqueName: \"kubernetes.io/projected/65537e68-a868-41a3-99a5-d6b7ec61ec9a-kube-api-access-d9vmm\") pod \"mt-curl-7764-93f252\" (UID: \"65537e68-a868-41a3-99a5-d6b7ec61ec9a\") " pod="opendatahub/mt-curl-7764-93f252"
Sep 09 22:03:14.138242 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:14.138147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-93f252"
Sep 09 22:03:14.413911 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:14.413882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7764-93f252"]
Sep 09 22:03:14.414821 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:03:14.414795    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65537e68_a868_41a3_99a5_d6b7ec61ec9a.slice/crio-7b8f0d036259da8eac1eb230b08a868f65f5e6fcec85724fb8ad0c1605bbb743 WatchSource:0}: Error finding container 7b8f0d036259da8eac1eb230b08a868f65f5e6fcec85724fb8ad0c1605bbb743: Status 404 returned error can't find the container with id 7b8f0d036259da8eac1eb230b08a868f65f5e6fcec85724fb8ad0c1605bbb743
Sep 09 22:03:15.301180 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:15.301145    2563 generic.go:356] "Generic (PLEG): container finished" podID="65537e68-a868-41a3-99a5-d6b7ec61ec9a" containerID="26093a4d477182f126f0b983570d4700e9afac545bebf58add88b19821d0836b" exitCode=0
Sep 09 22:03:15.301568 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:15.301235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7764-93f252" event={"ID":"65537e68-a868-41a3-99a5-d6b7ec61ec9a","Type":"ContainerDied","Data":"26093a4d477182f126f0b983570d4700e9afac545bebf58add88b19821d0836b"}
Sep 09 22:03:15.301568 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:15.301278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7764-93f252" event={"ID":"65537e68-a868-41a3-99a5-d6b7ec61ec9a","Type":"ContainerStarted","Data":"7b8f0d036259da8eac1eb230b08a868f65f5e6fcec85724fb8ad0c1605bbb743"}
Sep 09 22:03:16.442893 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:16.442858    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-93f252"
Sep 09 22:03:16.462925 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:16.462900    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7764-93f252_65537e68-a868-41a3-99a5-d6b7ec61ec9a/mt-curl-7764-93f252/0.log"
Sep 09 22:03:16.487483 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:16.487450    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7764-93f252"]
Sep 09 22:03:16.491180 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:16.491158    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7764-93f252"]
Sep 09 22:03:16.547216 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:16.547184    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9vmm\" (UniqueName: \"kubernetes.io/projected/65537e68-a868-41a3-99a5-d6b7ec61ec9a-kube-api-access-d9vmm\") pod \"65537e68-a868-41a3-99a5-d6b7ec61ec9a\" (UID: \"65537e68-a868-41a3-99a5-d6b7ec61ec9a\") "
Sep 09 22:03:16.549414 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:16.549384    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65537e68-a868-41a3-99a5-d6b7ec61ec9a-kube-api-access-d9vmm" (OuterVolumeSpecName: "kube-api-access-d9vmm") pod "65537e68-a868-41a3-99a5-d6b7ec61ec9a" (UID: "65537e68-a868-41a3-99a5-d6b7ec61ec9a"). InnerVolumeSpecName "kube-api-access-d9vmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:16.648926 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:16.648846    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9vmm\" (UniqueName: \"kubernetes.io/projected/65537e68-a868-41a3-99a5-d6b7ec61ec9a-kube-api-access-d9vmm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:17.314998 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:17.314967    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-93f252"
Sep 09 22:03:17.315174 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:17.314999    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b8f0d036259da8eac1eb230b08a868f65f5e6fcec85724fb8ad0c1605bbb743"
Sep 09 22:03:17.325743 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:03:17.325718    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7764-93f252\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="65537e68-a868-41a3-99a5-d6b7ec61ec9a" pod="opendatahub/mt-curl-7764-93f252"
Sep 09 22:03:17.606566 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:03:17.606464    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7764-93f252\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="65537e68-a868-41a3-99a5-d6b7ec61ec9a" pod="opendatahub/mt-curl-7764-93f252"
Sep 09 22:03:17.607752 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:17.607731    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65537e68-a868-41a3-99a5-d6b7ec61ec9a" path="/var/lib/kubelet/pods/65537e68-a868-41a3-99a5-d6b7ec61ec9a/volumes"
Sep 09 22:03:17.868701 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:17.868623    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"]
Sep 09 22:03:17.891686 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:17.891639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:17.932446 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:17.932385    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96" podUID="58a27e86-277c-4b11-8527-2a757d476d5a" containerName="main" containerID="cri-o://8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864" gracePeriod=45
Sep 09 22:03:18.191931 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.191906    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:03:18.322745 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.322711    2563 generic.go:356] "Generic (PLEG): container finished" podID="58a27e86-277c-4b11-8527-2a757d476d5a" containerID="8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864" exitCode=0
Sep 09 22:03:18.322963 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.322752    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"
Sep 09 22:03:18.322963 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.322809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96" event={"ID":"58a27e86-277c-4b11-8527-2a757d476d5a","Type":"ContainerDied","Data":"8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864"}
Sep 09 22:03:18.322963 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.322861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96" event={"ID":"58a27e86-277c-4b11-8527-2a757d476d5a","Type":"ContainerDied","Data":"b5199a9b519d0f5cca155f1ded28732564c25c7b4d0105eec6eaa4ab03ad4f05"}
Sep 09 22:03:18.322963 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.322884    2563 scope.go:117] "RemoveContainer" containerID="8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864"
Sep 09 22:03:18.332309 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.332293    2563 scope.go:117] "RemoveContainer" containerID="8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864"
Sep 09 22:03:18.332608 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:03:18.332585    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864\": container with ID starting with 8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864 not found: ID does not exist" containerID="8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864"
Sep 09 22:03:18.332654 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.332617    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864"} err="failed to get container status \"8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864\": rpc error: code = NotFound desc = could not find container \"8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864\": container with ID starting with 8e701812241d897f65592429b0ca2e3c0ceb082eced33e6dcd12072da4c0b864 not found: ID does not exist"
Sep 09 22:03:18.362982 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.362957    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-home\") pod \"58a27e86-277c-4b11-8527-2a757d476d5a\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") "
Sep 09 22:03:18.363093 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.363004    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sgtq\" (UniqueName: \"kubernetes.io/projected/58a27e86-277c-4b11-8527-2a757d476d5a-kube-api-access-2sgtq\") pod \"58a27e86-277c-4b11-8527-2a757d476d5a\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") "
Sep 09 22:03:18.363093 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.363038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a27e86-277c-4b11-8527-2a757d476d5a-tls-certs\") pod \"58a27e86-277c-4b11-8527-2a757d476d5a\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") "
Sep 09 22:03:18.363093 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.363085    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-tmp-dir\") pod \"58a27e86-277c-4b11-8527-2a757d476d5a\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") "
Sep 09 22:03:18.363252 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.363108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-dshm\") pod \"58a27e86-277c-4b11-8527-2a757d476d5a\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") "
Sep 09 22:03:18.363252 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.363144    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-model-cache\") pod \"58a27e86-277c-4b11-8527-2a757d476d5a\" (UID: \"58a27e86-277c-4b11-8527-2a757d476d5a\") "
Sep 09 22:03:18.363358 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.363262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-home" (OuterVolumeSpecName: "home") pod "58a27e86-277c-4b11-8527-2a757d476d5a" (UID: "58a27e86-277c-4b11-8527-2a757d476d5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:18.363358 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.363336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58a27e86-277c-4b11-8527-2a757d476d5a" (UID: "58a27e86-277c-4b11-8527-2a757d476d5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:18.363459 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.363435    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "58a27e86-277c-4b11-8527-2a757d476d5a" (UID: "58a27e86-277c-4b11-8527-2a757d476d5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:18.363459 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.363448    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.363538 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.363469    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.365294 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.365274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a27e86-277c-4b11-8527-2a757d476d5a-kube-api-access-2sgtq" (OuterVolumeSpecName: "kube-api-access-2sgtq") pod "58a27e86-277c-4b11-8527-2a757d476d5a" (UID: "58a27e86-277c-4b11-8527-2a757d476d5a"). InnerVolumeSpecName "kube-api-access-2sgtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:18.365372 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.365314    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-dshm" (OuterVolumeSpecName: "dshm") pod "58a27e86-277c-4b11-8527-2a757d476d5a" (UID: "58a27e86-277c-4b11-8527-2a757d476d5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:18.365456 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.365441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a27e86-277c-4b11-8527-2a757d476d5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58a27e86-277c-4b11-8527-2a757d476d5a" (UID: "58a27e86-277c-4b11-8527-2a757d476d5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:03:18.464921 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.464828    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a27e86-277c-4b11-8527-2a757d476d5a-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.464921 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.464860    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.464921 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.464872    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a27e86-277c-4b11-8527-2a757d476d5a-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.464921 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.464884    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sgtq\" (UniqueName: \"kubernetes.io/projected/58a27e86-277c-4b11-8527-2a757d476d5a-kube-api-access-2sgtq\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:18.645383 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.645350    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"]
Sep 09 22:03:18.649240 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.649215    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6a897d5c/e2e-lifecycle-model-6a897d5c-kserve-7b69449fcf-jwb96"]
Sep 09 22:03:18.963785 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.963733    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"]
Sep 09 22:03:18.974380 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:18.974346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:19.606305 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:19.606264    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58a27e86-277c-4b11-8527-2a757d476d5a" path="/var/lib/kubelet/pods/58a27e86-277c-4b11-8527-2a757d476d5a/volumes"
Sep 09 22:03:27.656980 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:27.656951    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 22:03:27.665983 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:27.665956    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 22:03:30.668554 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.668519    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"]
Sep 09 22:03:30.668976 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.668963    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a27e86-277c-4b11-8527-2a757d476d5a" containerName="main"
Sep 09 22:03:30.668976 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.668977    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a27e86-277c-4b11-8527-2a757d476d5a" containerName="main"
Sep 09 22:03:30.669069 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.668990    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65537e68-a868-41a3-99a5-d6b7ec61ec9a" containerName="mt-curl-7764-93f252"
Sep 09 22:03:30.669069 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.668996    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="65537e68-a868-41a3-99a5-d6b7ec61ec9a" containerName="mt-curl-7764-93f252"
Sep 09 22:03:30.669069 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.669068    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="65537e68-a868-41a3-99a5-d6b7ec61ec9a" containerName="mt-curl-7764-93f252"
Sep 09 22:03:30.669162 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.669076    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a27e86-277c-4b11-8527-2a757d476d5a" containerName="main"
Sep 09 22:03:30.673300 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.673280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.676136 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.676108    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ace629\"/\"sub-test-model-ace629-kserve-self-signed-certs\""
Sep 09 22:03:30.681843 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.681820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"]
Sep 09 22:03:30.694579 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.694553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:30.773612 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.773571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20920256-5d81-4b60-a218-05a32696d0bd-tls-certs\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.773807 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.773630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-model-cache\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.773807 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.773679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-tmp-dir\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.773807 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.773713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-home\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.773807 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.773736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-dshm\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.773960 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.773876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg2ch\" (UniqueName: \"kubernetes.io/projected/20920256-5d81-4b60-a218-05a32696d0bd-kube-api-access-wg2ch\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.874685 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.874647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg2ch\" (UniqueName: \"kubernetes.io/projected/20920256-5d81-4b60-a218-05a32696d0bd-kube-api-access-wg2ch\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.874872 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.874694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20920256-5d81-4b60-a218-05a32696d0bd-tls-certs\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.874872 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.874742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-model-cache\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.874872 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.874802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-tmp-dir\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.874872 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.874827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-home\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.874872 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.874854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-dshm\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.875255 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.875233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-tmp-dir\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.875324 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.875302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-home\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.875324 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.875315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-model-cache\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.877243 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.877221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-dshm\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.877368 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.877354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20920256-5d81-4b60-a218-05a32696d0bd-tls-certs\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.883459 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.883437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg2ch\" (UniqueName: \"kubernetes.io/projected/20920256-5d81-4b60-a218-05a32696d0bd-kube-api-access-wg2ch\") pod \"sub-test-model-ace629-kserve-588bc4944c-plz2c\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") " pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:30.984876 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:30.984790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:31.264595 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:31.264568    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"]
Sep 09 22:03:31.265369 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:03:31.265341    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20920256_5d81_4b60_a218_05a32696d0bd.slice/crio-e0a2774864b75ffd3c960231b7b48d4de30f78005bf55c524d879b58cb2d69dc WatchSource:0}: Error finding container e0a2774864b75ffd3c960231b7b48d4de30f78005bf55c524d879b58cb2d69dc: Status 404 returned error can't find the container with id e0a2774864b75ffd3c960231b7b48d4de30f78005bf55c524d879b58cb2d69dc
Sep 09 22:03:31.400377 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:31.400343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c" event={"ID":"20920256-5d81-4b60-a218-05a32696d0bd","Type":"ContainerStarted","Data":"75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e"}
Sep 09 22:03:31.400534 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:31.400384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c" event={"ID":"20920256-5d81-4b60-a218-05a32696d0bd","Type":"ContainerStarted","Data":"e0a2774864b75ffd3c960231b7b48d4de30f78005bf55c524d879b58cb2d69dc"}
Sep 09 22:03:31.416717 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:31.416647    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c" podStartSLOduration=1.416630653 podStartE2EDuration="1.416630653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:03:31.416185686 +0000 UTC m=+1504.312851695" watchObservedRunningTime="2026-09-09 22:03:31.416630653 +0000 UTC m=+1504.313296661"
Sep 09 22:03:32.889298 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:32.889237    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7" podUID="e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" containerName="main" containerID="cri-o://5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183" gracePeriod=45
Sep 09 22:03:33.148570 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.148512    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:03:33.199730 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.199695    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tls-certs\") pod \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") "
Sep 09 22:03:33.199730 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.199731    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-dshm\") pod \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") "
Sep 09 22:03:33.199983 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.199753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-model-cache\") pod \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") "
Sep 09 22:03:33.199983 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.199816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-home\") pod \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") "
Sep 09 22:03:33.199983 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.199882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m44ll\" (UniqueName: \"kubernetes.io/projected/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-kube-api-access-m44ll\") pod \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") "
Sep 09 22:03:33.199983 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.199948    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tmp-dir\") pod \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\" (UID: \"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7\") "
Sep 09 22:03:33.200195 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.200124    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" (UID: "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:33.200195 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.200158    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-home" (OuterVolumeSpecName: "home") pod "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" (UID: "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:33.200298 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.200241    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" (UID: "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:33.200349 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.200298    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:33.200349 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.200317    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:33.202171 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.202135    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-dshm" (OuterVolumeSpecName: "dshm") pod "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" (UID: "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:33.202290 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.202190    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" (UID: "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:03:33.202290 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.202217    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-kube-api-access-m44ll" (OuterVolumeSpecName: "kube-api-access-m44ll") pod "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" (UID: "e6d7d803-7ce7-4f03-900e-fa96da4fe0b7"). InnerVolumeSpecName "kube-api-access-m44ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:33.301515 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.301485    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:33.301515 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.301512    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:33.301515 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.301523    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:33.301760 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.301532    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m44ll\" (UniqueName: \"kubernetes.io/projected/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7-kube-api-access-m44ll\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:33.412897 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.412815    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" containerID="5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183" exitCode=0
Sep 09 22:03:33.412897 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.412857    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"
Sep 09 22:03:33.413076 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.412893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7" event={"ID":"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7","Type":"ContainerDied","Data":"5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183"}
Sep 09 22:03:33.413076 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.412940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7" event={"ID":"e6d7d803-7ce7-4f03-900e-fa96da4fe0b7","Type":"ContainerDied","Data":"dd8c80cafa9c0a465a1d30834d3df1c0b12ab5b9cccd31012cbffa242dfa4177"}
Sep 09 22:03:33.413076 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.412958    2563 scope.go:117] "RemoveContainer" containerID="5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183"
Sep 09 22:03:33.422941 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.422926    2563 scope.go:117] "RemoveContainer" containerID="5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183"
Sep 09 22:03:33.423204 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:03:33.423187    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183\": container with ID starting with 5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183 not found: ID does not exist" containerID="5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183"
Sep 09 22:03:33.423246 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.423214    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183"} err="failed to get container status \"5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183\": rpc error: code = NotFound desc = could not find container \"5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183\": container with ID starting with 5f6b01205389ec5bf9fefd8cb50c3e3115eb6982ae01cbb03de3db55aadff183 not found: ID does not exist"
Sep 09 22:03:33.437462 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.437442    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"]
Sep 09 22:03:33.441660 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.441638    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/auth-test-model-ace629-kserve-645949d659-4gqz7"]
Sep 09 22:03:33.606491 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.606458    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" path="/var/lib/kubelet/pods/e6d7d803-7ce7-4f03-900e-fa96da4fe0b7/volumes"
Sep 09 22:03:33.984647 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:33.984602    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll" podUID="78a6f377-acde-4c50-a2e1-3526b51071e1" containerName="main" containerID="cri-o://7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b" gracePeriod=45
Sep 09 22:03:34.249031 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.249004    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:03:34.310286 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310246    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-tmp-dir\") pod \"78a6f377-acde-4c50-a2e1-3526b51071e1\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") "
Sep 09 22:03:34.310467 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-dshm\") pod \"78a6f377-acde-4c50-a2e1-3526b51071e1\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") "
Sep 09 22:03:34.310467 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310353    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkdbx\" (UniqueName: \"kubernetes.io/projected/78a6f377-acde-4c50-a2e1-3526b51071e1-kube-api-access-zkdbx\") pod \"78a6f377-acde-4c50-a2e1-3526b51071e1\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") "
Sep 09 22:03:34.310467 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310390    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a6f377-acde-4c50-a2e1-3526b51071e1-tls-certs\") pod \"78a6f377-acde-4c50-a2e1-3526b51071e1\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") "
Sep 09 22:03:34.310467 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-home\") pod \"78a6f377-acde-4c50-a2e1-3526b51071e1\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") "
Sep 09 22:03:34.310467 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310456    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-model-cache\") pod \"78a6f377-acde-4c50-a2e1-3526b51071e1\" (UID: \"78a6f377-acde-4c50-a2e1-3526b51071e1\") "
Sep 09 22:03:34.310740 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78a6f377-acde-4c50-a2e1-3526b51071e1" (UID: "78a6f377-acde-4c50-a2e1-3526b51071e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:34.310740 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310690    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-home" (OuterVolumeSpecName: "home") pod "78a6f377-acde-4c50-a2e1-3526b51071e1" (UID: "78a6f377-acde-4c50-a2e1-3526b51071e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:34.310880 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310731    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "78a6f377-acde-4c50-a2e1-3526b51071e1" (UID: "78a6f377-acde-4c50-a2e1-3526b51071e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:34.310880 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310823    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:34.310880 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.310837    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:34.312880 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.312859    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a6f377-acde-4c50-a2e1-3526b51071e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78a6f377-acde-4c50-a2e1-3526b51071e1" (UID: "78a6f377-acde-4c50-a2e1-3526b51071e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:03:34.313129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.313105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a6f377-acde-4c50-a2e1-3526b51071e1-kube-api-access-zkdbx" (OuterVolumeSpecName: "kube-api-access-zkdbx") pod "78a6f377-acde-4c50-a2e1-3526b51071e1" (UID: "78a6f377-acde-4c50-a2e1-3526b51071e1"). InnerVolumeSpecName "kube-api-access-zkdbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:34.313129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.313107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-dshm" (OuterVolumeSpecName: "dshm") pod "78a6f377-acde-4c50-a2e1-3526b51071e1" (UID: "78a6f377-acde-4c50-a2e1-3526b51071e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:34.412172 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.412129    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a6f377-acde-4c50-a2e1-3526b51071e1-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:34.412172 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.412164    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:34.412172 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.412173    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a6f377-acde-4c50-a2e1-3526b51071e1-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:34.412172 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.412183    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkdbx\" (UniqueName: \"kubernetes.io/projected/78a6f377-acde-4c50-a2e1-3526b51071e1-kube-api-access-zkdbx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:03:34.421720 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.421690    2563 generic.go:356] "Generic (PLEG): container finished" podID="78a6f377-acde-4c50-a2e1-3526b51071e1" containerID="7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b" exitCode=0
Sep 09 22:03:34.421880 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.421735    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"
Sep 09 22:03:34.421880 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.421789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll" event={"ID":"78a6f377-acde-4c50-a2e1-3526b51071e1","Type":"ContainerDied","Data":"7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b"}
Sep 09 22:03:34.421880 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.421831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll" event={"ID":"78a6f377-acde-4c50-a2e1-3526b51071e1","Type":"ContainerDied","Data":"ae25c034f29308ccd51016034c3153d22dd4749b75d169f01d190cdc60c534a0"}
Sep 09 22:03:34.421880 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.421849    2563 scope.go:117] "RemoveContainer" containerID="7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b"
Sep 09 22:03:34.431235 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.431214    2563 scope.go:117] "RemoveContainer" containerID="7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b"
Sep 09 22:03:34.431511 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:03:34.431493    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b\": container with ID starting with 7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b not found: ID does not exist" containerID="7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b"
Sep 09 22:03:34.431555 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.431522    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b"} err="failed to get container status \"7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b\": rpc error: code = NotFound desc = could not find container \"7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b\": container with ID starting with 7ff31bd5530c093f1acf1a0d308fc99f206809391d973e32cd838fc0e6493c5b not found: ID does not exist"
Sep 09 22:03:34.443421 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.443397    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"]
Sep 09 22:03:34.447729 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:34.447707    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/auth-test-model-3b2190-kserve-86f8f7c75f-f9dll"]
Sep 09 22:03:35.607224 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:35.607195    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a6f377-acde-4c50-a2e1-3526b51071e1" path="/var/lib/kubelet/pods/78a6f377-acde-4c50-a2e1-3526b51071e1/volumes"
Sep 09 22:03:39.153730 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:39.153699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:40.207661 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:40.207627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:40.985613 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:40.985570    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:40.985825 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:40.985646    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:40.998808 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:40.998762    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:41.475896 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:41.475869    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:03:45.842115 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:03:45.842079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:18.044704 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:18.044620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:18.228031 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:18.227995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:18.406518 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:18.406433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:19.671729 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:19.671690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:19.766631 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:19.766599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:19.864986 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:19.864953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:20.456986 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:20.456947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:20.554384 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:20.554356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:20.655260 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:20.655176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:21.902374 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:21.902221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:26.271290 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.271252    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"]
Sep 09 22:04:26.271742 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.271727    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" containerName="main"
Sep 09 22:04:26.271807 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.271743    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" containerName="main"
Sep 09 22:04:26.271807 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.271761    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a6f377-acde-4c50-a2e1-3526b51071e1" containerName="main"
Sep 09 22:04:26.271807 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.271785    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a6f377-acde-4c50-a2e1-3526b51071e1" containerName="main"
Sep 09 22:04:26.271905 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.271865    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6d7d803-7ce7-4f03-900e-fa96da4fe0b7" containerName="main"
Sep 09 22:04:26.271905 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.271875    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a6f377-acde-4c50-a2e1-3526b51071e1" containerName="main"
Sep 09 22:04:26.275074 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.275056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.277829 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.277807    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"openshift-service-ca.crt\""
Sep 09 22:04:26.277925 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.277852    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"default-dockercfg-pkxqr\""
Sep 09 22:04:26.278977 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.278954    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"kube-root-ca.crt\""
Sep 09 22:04:26.279080 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.279064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"sub-test-model-3b2190-kserve-self-signed-certs\""
Sep 09 22:04:26.284663 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.284638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"]
Sep 09 22:04:26.356886 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.356846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:26.400635 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.400600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faf582a-0e2a-4714-a55a-2741eee7cf32-tls-certs\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.400835 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.400643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-home\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.400835 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.400665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-dshm\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.400835 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.400755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4t9\" (UniqueName: \"kubernetes.io/projected/5faf582a-0e2a-4714-a55a-2741eee7cf32-kube-api-access-bs4t9\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.400835 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.400816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-tmp-dir\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.400992 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.400842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-model-cache\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.501341 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.501301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-dshm\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.501550 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.501352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4t9\" (UniqueName: \"kubernetes.io/projected/5faf582a-0e2a-4714-a55a-2741eee7cf32-kube-api-access-bs4t9\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.501550 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.501397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-tmp-dir\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.501550 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.501422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-model-cache\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.501734 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.501576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faf582a-0e2a-4714-a55a-2741eee7cf32-tls-certs\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.501734 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.501644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-home\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.501909 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.501884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-tmp-dir\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.501909 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.501905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-model-cache\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.502016 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.501984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-home\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.503919 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.503893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-dshm\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.504117 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.504099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faf582a-0e2a-4714-a55a-2741eee7cf32-tls-certs\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.513459 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.513434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4t9\" (UniqueName: \"kubernetes.io/projected/5faf582a-0e2a-4714-a55a-2741eee7cf32-kube-api-access-bs4t9\") pod \"sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.585129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.585031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:26.885112 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:26.885084    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"]
Sep 09 22:04:26.890490 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:04:26.890414    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5faf582a_0e2a_4714_a55a_2741eee7cf32.slice/crio-befb8a2cc7352dc6ae3718cdc6c9e2755f02f6461202b3f328bc41ac8137ed7d WatchSource:0}: Error finding container befb8a2cc7352dc6ae3718cdc6c9e2755f02f6461202b3f328bc41ac8137ed7d: Status 404 returned error can't find the container with id befb8a2cc7352dc6ae3718cdc6c9e2755f02f6461202b3f328bc41ac8137ed7d
Sep 09 22:04:27.738628 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:27.738591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9" event={"ID":"5faf582a-0e2a-4714-a55a-2741eee7cf32","Type":"ContainerStarted","Data":"78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e"}
Sep 09 22:04:27.738628 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:27.738630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9" event={"ID":"5faf582a-0e2a-4714-a55a-2741eee7cf32","Type":"ContainerStarted","Data":"befb8a2cc7352dc6ae3718cdc6c9e2755f02f6461202b3f328bc41ac8137ed7d"}
Sep 09 22:04:27.757282 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:27.757234    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9" podStartSLOduration=1.7572179700000001 podStartE2EDuration="1.75721797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:04:27.756411585 +0000 UTC m=+1560.653077594" watchObservedRunningTime="2026-09-09 22:04:27.75721797 +0000 UTC m=+1560.653883977"
Sep 09 22:04:28.122429 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:28.122399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:28.224168 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:28.224118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:28.786428 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:28.786395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:34.597128 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:34.597092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:34.705508 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:34.705479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:34.801990 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:34.801962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:36.585467 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:36.585436    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:36.585919 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:36.585487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:36.598099 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:36.598074    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:36.804500 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:36.804471    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:04:43.778122 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:43.778084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:45.279303 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:45.279264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:45.782362 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:45.782333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:59.060030 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.059999    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7764-d50807"]
Sep 09 22:04:59.063589 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.063571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-d50807"
Sep 09 22:04:59.066719 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.066684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fkr9w\""
Sep 09 22:04:59.069598 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.069554    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7764-d50807"]
Sep 09 22:04:59.094053 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.094028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4sh\" (UniqueName: \"kubernetes.io/projected/5fd37f95-2c2f-4932-9477-ddae3e168af8-kube-api-access-5s4sh\") pod \"mt-curl-7764-d50807\" (UID: \"5fd37f95-2c2f-4932-9477-ddae3e168af8\") " pod="opendatahub/mt-curl-7764-d50807"
Sep 09 22:04:59.195132 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.195096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4sh\" (UniqueName: \"kubernetes.io/projected/5fd37f95-2c2f-4932-9477-ddae3e168af8-kube-api-access-5s4sh\") pod \"mt-curl-7764-d50807\" (UID: \"5fd37f95-2c2f-4932-9477-ddae3e168af8\") " pod="opendatahub/mt-curl-7764-d50807"
Sep 09 22:04:59.203913 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.203877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4sh\" (UniqueName: \"kubernetes.io/projected/5fd37f95-2c2f-4932-9477-ddae3e168af8-kube-api-access-5s4sh\") pod \"mt-curl-7764-d50807\" (UID: \"5fd37f95-2c2f-4932-9477-ddae3e168af8\") " pod="opendatahub/mt-curl-7764-d50807"
Sep 09 22:04:59.374380 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.374293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-d50807"
Sep 09 22:04:59.645006 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.644982    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7764-d50807"]
Sep 09 22:04:59.647518 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:04:59.647493    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fd37f95_2c2f_4932_9477_ddae3e168af8.slice/crio-f5984aaaa1f15d44e66dd78610243e483690fef12dcafdcd3866e0fec22505b7 WatchSource:0}: Error finding container f5984aaaa1f15d44e66dd78610243e483690fef12dcafdcd3866e0fec22505b7: Status 404 returned error can't find the container with id f5984aaaa1f15d44e66dd78610243e483690fef12dcafdcd3866e0fec22505b7
Sep 09 22:04:59.648829 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.648812    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 22:04:59.932801 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:04:59.932779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7764-d50807" event={"ID":"5fd37f95-2c2f-4932-9477-ddae3e168af8","Type":"ContainerStarted","Data":"f5984aaaa1f15d44e66dd78610243e483690fef12dcafdcd3866e0fec22505b7"}
Sep 09 22:05:00.940504 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:00.940468    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fd37f95-2c2f-4932-9477-ddae3e168af8" containerID="6c599f884ebca4c4a0fdb4ad87139f8981009229ef6efe0f2f4e844d147979f2" exitCode=0
Sep 09 22:05:00.940924 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:00.940552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7764-d50807" event={"ID":"5fd37f95-2c2f-4932-9477-ddae3e168af8","Type":"ContainerDied","Data":"6c599f884ebca4c4a0fdb4ad87139f8981009229ef6efe0f2f4e844d147979f2"}
Sep 09 22:05:02.083810 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.083782    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-d50807"
Sep 09 22:05:02.105050 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.105018    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7764-d50807_5fd37f95-2c2f-4932-9477-ddae3e168af8/mt-curl-7764-d50807/0.log"
Sep 09 22:05:02.120372 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.120337    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s4sh\" (UniqueName: \"kubernetes.io/projected/5fd37f95-2c2f-4932-9477-ddae3e168af8-kube-api-access-5s4sh\") pod \"5fd37f95-2c2f-4932-9477-ddae3e168af8\" (UID: \"5fd37f95-2c2f-4932-9477-ddae3e168af8\") "
Sep 09 22:05:02.122590 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.122565    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd37f95-2c2f-4932-9477-ddae3e168af8-kube-api-access-5s4sh" (OuterVolumeSpecName: "kube-api-access-5s4sh") pod "5fd37f95-2c2f-4932-9477-ddae3e168af8" (UID: "5fd37f95-2c2f-4932-9477-ddae3e168af8"). InnerVolumeSpecName "kube-api-access-5s4sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:05:02.130197 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.130172    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7764-d50807"]
Sep 09 22:05:02.134052 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.134022    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7764-d50807"]
Sep 09 22:05:02.221363 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.221273    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s4sh\" (UniqueName: \"kubernetes.io/projected/5fd37f95-2c2f-4932-9477-ddae3e168af8-kube-api-access-5s4sh\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:02.249784 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.249738    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7764-8a25ba"]
Sep 09 22:05:02.250191 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.250177    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd37f95-2c2f-4932-9477-ddae3e168af8" containerName="mt-curl-7764-d50807"
Sep 09 22:05:02.250241 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.250192    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd37f95-2c2f-4932-9477-ddae3e168af8" containerName="mt-curl-7764-d50807"
Sep 09 22:05:02.250278 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.250263    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd37f95-2c2f-4932-9477-ddae3e168af8" containerName="mt-curl-7764-d50807"
Sep 09 22:05:02.253264 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.253247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-8a25ba"
Sep 09 22:05:02.261618 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.261345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7764-8a25ba"]
Sep 09 22:05:02.322451 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.322411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxsjg\" (UniqueName: \"kubernetes.io/projected/b7199e19-fb78-48b5-b5b8-5d528444edef-kube-api-access-wxsjg\") pod \"mt-curl-7764-8a25ba\" (UID: \"b7199e19-fb78-48b5-b5b8-5d528444edef\") " pod="opendatahub/mt-curl-7764-8a25ba"
Sep 09 22:05:02.422941 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.422904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsjg\" (UniqueName: \"kubernetes.io/projected/b7199e19-fb78-48b5-b5b8-5d528444edef-kube-api-access-wxsjg\") pod \"mt-curl-7764-8a25ba\" (UID: \"b7199e19-fb78-48b5-b5b8-5d528444edef\") " pod="opendatahub/mt-curl-7764-8a25ba"
Sep 09 22:05:02.431898 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.431872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxsjg\" (UniqueName: \"kubernetes.io/projected/b7199e19-fb78-48b5-b5b8-5d528444edef-kube-api-access-wxsjg\") pod \"mt-curl-7764-8a25ba\" (UID: \"b7199e19-fb78-48b5-b5b8-5d528444edef\") " pod="opendatahub/mt-curl-7764-8a25ba"
Sep 09 22:05:02.564917 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.564885    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-8a25ba"
Sep 09 22:05:02.844497 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.844469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7764-8a25ba"]
Sep 09 22:05:02.845537 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:05:02.845511    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7199e19_fb78_48b5_b5b8_5d528444edef.slice/crio-ca04e422c23604ea4b4a880e45e948f4cba5c67aa4af68ee131d2065f115b5d5 WatchSource:0}: Error finding container ca04e422c23604ea4b4a880e45e948f4cba5c67aa4af68ee131d2065f115b5d5: Status 404 returned error can't find the container with id ca04e422c23604ea4b4a880e45e948f4cba5c67aa4af68ee131d2065f115b5d5
Sep 09 22:05:02.954505 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.954478    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-d50807"
Sep 09 22:05:02.954711 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.954510    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5984aaaa1f15d44e66dd78610243e483690fef12dcafdcd3866e0fec22505b7"
Sep 09 22:05:02.955936 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:02.955891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7764-8a25ba" event={"ID":"b7199e19-fb78-48b5-b5b8-5d528444edef","Type":"ContainerStarted","Data":"ca04e422c23604ea4b4a880e45e948f4cba5c67aa4af68ee131d2065f115b5d5"}
Sep 09 22:05:02.965320 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:05:02.965293    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7764-d50807\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="5fd37f95-2c2f-4932-9477-ddae3e168af8" pod="opendatahub/mt-curl-7764-d50807"
Sep 09 22:05:03.607412 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:03.607376    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd37f95-2c2f-4932-9477-ddae3e168af8" path="/var/lib/kubelet/pods/5fd37f95-2c2f-4932-9477-ddae3e168af8/volumes"
Sep 09 22:05:03.963686 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:03.963594    2563 generic.go:356] "Generic (PLEG): container finished" podID="b7199e19-fb78-48b5-b5b8-5d528444edef" containerID="be68af4cafcbb34fb21754808ae362d9cde529e282bd105f4dce1bc351bfb7bc" exitCode=0
Sep 09 22:05:03.963686 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:03.963645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7764-8a25ba" event={"ID":"b7199e19-fb78-48b5-b5b8-5d528444edef","Type":"ContainerDied","Data":"be68af4cafcbb34fb21754808ae362d9cde529e282bd105f4dce1bc351bfb7bc"}
Sep 09 22:05:05.103566 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:05.103539    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-8a25ba"
Sep 09 22:05:05.125016 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:05.124989    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7764-8a25ba_b7199e19-fb78-48b5-b5b8-5d528444edef/mt-curl-7764-8a25ba/0.log"
Sep 09 22:05:05.147542 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:05.147507    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7764-8a25ba"]
Sep 09 22:05:05.150192 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:05.150165    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxsjg\" (UniqueName: \"kubernetes.io/projected/b7199e19-fb78-48b5-b5b8-5d528444edef-kube-api-access-wxsjg\") pod \"b7199e19-fb78-48b5-b5b8-5d528444edef\" (UID: \"b7199e19-fb78-48b5-b5b8-5d528444edef\") "
Sep 09 22:05:05.151747 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:05.151714    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7764-8a25ba"]
Sep 09 22:05:05.152482 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:05.152456    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7199e19-fb78-48b5-b5b8-5d528444edef-kube-api-access-wxsjg" (OuterVolumeSpecName: "kube-api-access-wxsjg") pod "b7199e19-fb78-48b5-b5b8-5d528444edef" (UID: "b7199e19-fb78-48b5-b5b8-5d528444edef"). InnerVolumeSpecName "kube-api-access-wxsjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:05:05.251267 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:05.251236    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxsjg\" (UniqueName: \"kubernetes.io/projected/b7199e19-fb78-48b5-b5b8-5d528444edef-kube-api-access-wxsjg\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:05.607546 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:05.607463    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7199e19-fb78-48b5-b5b8-5d528444edef" path="/var/lib/kubelet/pods/b7199e19-fb78-48b5-b5b8-5d528444edef/volumes"
Sep 09 22:05:05.976554 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:05.976466    2563 scope.go:117] "RemoveContainer" containerID="be68af4cafcbb34fb21754808ae362d9cde529e282bd105f4dce1bc351bfb7bc"
Sep 09 22:05:05.976554 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:05.976467    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7764-8a25ba"
Sep 09 22:05:06.426508 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:06.426475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:07.167052 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:07.167015    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"]
Sep 09 22:05:07.190739 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:07.190688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:08.164498 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:08.164467    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"]
Sep 09 22:05:08.191895 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:08.191857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:12.945941 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:12.945911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:13.046452 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:13.046418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:13.221232 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:13.221147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:13.584566 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:13.584534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:16.157741 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:16.157697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:16.247324 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:16.247290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:17.279201 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.279166    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"]
Sep 09 22:05:17.279855 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.279838    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7199e19-fb78-48b5-b5b8-5d528444edef" containerName="mt-curl-7764-8a25ba"
Sep 09 22:05:17.279901 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.279859    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7199e19-fb78-48b5-b5b8-5d528444edef" containerName="mt-curl-7764-8a25ba"
Sep 09 22:05:17.279985 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.279974    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7199e19-fb78-48b5-b5b8-5d528444edef" containerName="mt-curl-7764-8a25ba"
Sep 09 22:05:17.285445 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.285420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.289800 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.289752    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8a52f0-serving-cert\""
Sep 09 22:05:17.293212 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.293187    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"]
Sep 09 22:05:17.318436 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.318409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:17.368626 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.368591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-maas-api-tls\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.368840 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.368682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b34d7f-f861-4066-bef8-df9514e9afce-openshift-service-ca\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.368840 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.368705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-metrics-certs\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.368840 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.368804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6fm\" (UniqueName: \"kubernetes.io/projected/a4b34d7f-f861-4066-bef8-df9514e9afce-kube-api-access-6l6fm\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.434932 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.434903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:17.470050 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.470019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b34d7f-f861-4066-bef8-df9514e9afce-openshift-service-ca\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.470050 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.470053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-metrics-certs\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.470292 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.470109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6fm\" (UniqueName: \"kubernetes.io/projected/a4b34d7f-f861-4066-bef8-df9514e9afce-kube-api-access-6l6fm\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.470292 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.470141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-maas-api-tls\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.470741 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.470718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b34d7f-f861-4066-bef8-df9514e9afce-openshift-service-ca\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.472658 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.472637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-maas-api-tls\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.472794 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.472752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-metrics-certs\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.478908 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.478886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6fm\" (UniqueName: \"kubernetes.io/projected/a4b34d7f-f861-4066-bef8-df9514e9afce-kube-api-access-6l6fm\") pod \"maas-api-e2e-api-a-8a52f0-5584798775-h55pl\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.518759 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.518727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:17.599802 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.599678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:17.833678 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.833644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:17.884377 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:17.884342    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"]
Sep 09 22:05:17.885662 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:05:17.885636    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b34d7f_f861_4066_bef8_df9514e9afce.slice/crio-b3cac655ccd8429d2978d1d922f03774a9706e4a2a2e25c5590de8909d745e2b WatchSource:0}: Error finding container b3cac655ccd8429d2978d1d922f03774a9706e4a2a2e25c5590de8909d745e2b: Status 404 returned error can't find the container with id b3cac655ccd8429d2978d1d922f03774a9706e4a2a2e25c5590de8909d745e2b
Sep 09 22:05:18.015363 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:18.015330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:18.054966 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:18.054935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl" event={"ID":"a4b34d7f-f861-4066-bef8-df9514e9afce","Type":"ContainerStarted","Data":"b3cac655ccd8429d2978d1d922f03774a9706e4a2a2e25c5590de8909d745e2b"}
Sep 09 22:05:19.064535 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:19.064488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl" event={"ID":"a4b34d7f-f861-4066-bef8-df9514e9afce","Type":"ContainerStarted","Data":"e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e"}
Sep 09 22:05:19.065048 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:19.064686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:19.082724 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:19.082666    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl" podStartSLOduration=1.258589159 podStartE2EDuration="2.082648202s" totalImagesPullingTime="824.059043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:19.081109752 +0000 UTC m=+1611.977775759" watchObservedRunningTime="2026-09-09 22:05:19.082648202 +0000 UTC m=+1611.979314209"
Sep 09 22:05:22.188313 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.188255    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c" podUID="20920256-5d81-4b60-a218-05a32696d0bd" containerName="main" containerID="cri-o://75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e" gracePeriod=45
Sep 09 22:05:22.489841 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.486938    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:05:22.613647 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.613609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-home\") pod \"20920256-5d81-4b60-a218-05a32696d0bd\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") "
Sep 09 22:05:22.613865 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.613656    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-dshm\") pod \"20920256-5d81-4b60-a218-05a32696d0bd\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") "
Sep 09 22:05:22.613865 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.613696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-tmp-dir\") pod \"20920256-5d81-4b60-a218-05a32696d0bd\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") "
Sep 09 22:05:22.613865 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.613739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20920256-5d81-4b60-a218-05a32696d0bd-tls-certs\") pod \"20920256-5d81-4b60-a218-05a32696d0bd\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") "
Sep 09 22:05:22.613865 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.613795    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-model-cache\") pod \"20920256-5d81-4b60-a218-05a32696d0bd\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") "
Sep 09 22:05:22.614162 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.613907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg2ch\" (UniqueName: \"kubernetes.io/projected/20920256-5d81-4b60-a218-05a32696d0bd-kube-api-access-wg2ch\") pod \"20920256-5d81-4b60-a218-05a32696d0bd\" (UID: \"20920256-5d81-4b60-a218-05a32696d0bd\") "
Sep 09 22:05:22.614162 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.613924    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-home" (OuterVolumeSpecName: "home") pod "20920256-5d81-4b60-a218-05a32696d0bd" (UID: "20920256-5d81-4b60-a218-05a32696d0bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:22.614162 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.613942    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20920256-5d81-4b60-a218-05a32696d0bd" (UID: "20920256-5d81-4b60-a218-05a32696d0bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:22.614162 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.614100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "20920256-5d81-4b60-a218-05a32696d0bd" (UID: "20920256-5d81-4b60-a218-05a32696d0bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:22.614440 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.614202    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.614440 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.614217    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.614440 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.614231    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.616280 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.616244    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-dshm" (OuterVolumeSpecName: "dshm") pod "20920256-5d81-4b60-a218-05a32696d0bd" (UID: "20920256-5d81-4b60-a218-05a32696d0bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:22.616420 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.616393    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20920256-5d81-4b60-a218-05a32696d0bd-kube-api-access-wg2ch" (OuterVolumeSpecName: "kube-api-access-wg2ch") pod "20920256-5d81-4b60-a218-05a32696d0bd" (UID: "20920256-5d81-4b60-a218-05a32696d0bd"). InnerVolumeSpecName "kube-api-access-wg2ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:05:22.616482 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.616396    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20920256-5d81-4b60-a218-05a32696d0bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20920256-5d81-4b60-a218-05a32696d0bd" (UID: "20920256-5d81-4b60-a218-05a32696d0bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:05:22.716052 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.716019    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg2ch\" (UniqueName: \"kubernetes.io/projected/20920256-5d81-4b60-a218-05a32696d0bd-kube-api-access-wg2ch\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.716052 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.716050    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20920256-5d81-4b60-a218-05a32696d0bd-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:22.716267 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:22.716065    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20920256-5d81-4b60-a218-05a32696d0bd-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:23.090498 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.090462    2563 generic.go:356] "Generic (PLEG): container finished" podID="20920256-5d81-4b60-a218-05a32696d0bd" containerID="75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e" exitCode=0
Sep 09 22:05:23.090709 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.090505    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"
Sep 09 22:05:23.090709 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.090551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c" event={"ID":"20920256-5d81-4b60-a218-05a32696d0bd","Type":"ContainerDied","Data":"75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e"}
Sep 09 22:05:23.090709 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.090588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c" event={"ID":"20920256-5d81-4b60-a218-05a32696d0bd","Type":"ContainerDied","Data":"e0a2774864b75ffd3c960231b7b48d4de30f78005bf55c524d879b58cb2d69dc"}
Sep 09 22:05:23.090709 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.090606    2563 scope.go:117] "RemoveContainer" containerID="75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e"
Sep 09 22:05:23.107290 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.107266    2563 scope.go:117] "RemoveContainer" containerID="75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e"
Sep 09 22:05:23.107712 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:05:23.107687    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e\": container with ID starting with 75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e not found: ID does not exist" containerID="75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e"
Sep 09 22:05:23.107834 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.107724    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e"} err="failed to get container status \"75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e\": rpc error: code = NotFound desc = could not find container \"75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e\": container with ID starting with 75d5ee2bde428617a44885da10b7a7ad7d3e04fc4e437281e7d005eed2e7b97e not found: ID does not exist"
Sep 09 22:05:23.117731 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.117708    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"]
Sep 09 22:05:23.121879 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.121850    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/sub-test-model-ace629-kserve-588bc4944c-plz2c"]
Sep 09 22:05:23.184991 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.184955    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9" podUID="5faf582a-0e2a-4714-a55a-2741eee7cf32" containerName="main" containerID="cri-o://78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e" gracePeriod=45
Sep 09 22:05:23.453622 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.453597    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:05:23.525505 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.525465    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faf582a-0e2a-4714-a55a-2741eee7cf32-tls-certs\") pod \"5faf582a-0e2a-4714-a55a-2741eee7cf32\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") "
Sep 09 22:05:23.525700 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.525562    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs4t9\" (UniqueName: \"kubernetes.io/projected/5faf582a-0e2a-4714-a55a-2741eee7cf32-kube-api-access-bs4t9\") pod \"5faf582a-0e2a-4714-a55a-2741eee7cf32\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") "
Sep 09 22:05:23.525700 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.525608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-tmp-dir\") pod \"5faf582a-0e2a-4714-a55a-2741eee7cf32\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") "
Sep 09 22:05:23.525700 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.525637    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-model-cache\") pod \"5faf582a-0e2a-4714-a55a-2741eee7cf32\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") "
Sep 09 22:05:23.525700 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.525696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-dshm\") pod \"5faf582a-0e2a-4714-a55a-2741eee7cf32\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") "
Sep 09 22:05:23.525970 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.525735    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-home\") pod \"5faf582a-0e2a-4714-a55a-2741eee7cf32\" (UID: \"5faf582a-0e2a-4714-a55a-2741eee7cf32\") "
Sep 09 22:05:23.525970 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.525925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5faf582a-0e2a-4714-a55a-2741eee7cf32" (UID: "5faf582a-0e2a-4714-a55a-2741eee7cf32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:23.525970 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.525940    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-model-cache" (OuterVolumeSpecName: "model-cache") pod "5faf582a-0e2a-4714-a55a-2741eee7cf32" (UID: "5faf582a-0e2a-4714-a55a-2741eee7cf32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:23.526106 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.526061    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-home" (OuterVolumeSpecName: "home") pod "5faf582a-0e2a-4714-a55a-2741eee7cf32" (UID: "5faf582a-0e2a-4714-a55a-2741eee7cf32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:23.526227 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.526209    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:23.526295 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.526251    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:23.526295 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.526261    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:23.527993 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.527964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-dshm" (OuterVolumeSpecName: "dshm") pod "5faf582a-0e2a-4714-a55a-2741eee7cf32" (UID: "5faf582a-0e2a-4714-a55a-2741eee7cf32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:23.527993 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.527977    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faf582a-0e2a-4714-a55a-2741eee7cf32-kube-api-access-bs4t9" (OuterVolumeSpecName: "kube-api-access-bs4t9") pod "5faf582a-0e2a-4714-a55a-2741eee7cf32" (UID: "5faf582a-0e2a-4714-a55a-2741eee7cf32"). InnerVolumeSpecName "kube-api-access-bs4t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:05:23.527993 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.527976    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5faf582a-0e2a-4714-a55a-2741eee7cf32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5faf582a-0e2a-4714-a55a-2741eee7cf32" (UID: "5faf582a-0e2a-4714-a55a-2741eee7cf32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:05:23.606677 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.606592    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20920256-5d81-4b60-a218-05a32696d0bd" path="/var/lib/kubelet/pods/20920256-5d81-4b60-a218-05a32696d0bd/volumes"
Sep 09 22:05:23.626749 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.626720    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faf582a-0e2a-4714-a55a-2741eee7cf32-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:23.626749 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.626749    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs4t9\" (UniqueName: \"kubernetes.io/projected/5faf582a-0e2a-4714-a55a-2741eee7cf32-kube-api-access-bs4t9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:23.626936 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:23.626759    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faf582a-0e2a-4714-a55a-2741eee7cf32-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:05:24.098320 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:24.098282    2563 generic.go:356] "Generic (PLEG): container finished" podID="5faf582a-0e2a-4714-a55a-2741eee7cf32" containerID="78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e" exitCode=0
Sep 09 22:05:24.098503 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:24.098328    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"
Sep 09 22:05:24.098503 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:24.098367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9" event={"ID":"5faf582a-0e2a-4714-a55a-2741eee7cf32","Type":"ContainerDied","Data":"78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e"}
Sep 09 22:05:24.098503 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:24.098412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9" event={"ID":"5faf582a-0e2a-4714-a55a-2741eee7cf32","Type":"ContainerDied","Data":"befb8a2cc7352dc6ae3718cdc6c9e2755f02f6461202b3f328bc41ac8137ed7d"}
Sep 09 22:05:24.098503 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:24.098430    2563 scope.go:117] "RemoveContainer" containerID="78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e"
Sep 09 22:05:24.107959 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:24.107943    2563 scope.go:117] "RemoveContainer" containerID="78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e"
Sep 09 22:05:24.108228 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:05:24.108209    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e\": container with ID starting with 78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e not found: ID does not exist" containerID="78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e"
Sep 09 22:05:24.108301 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:24.108235    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e"} err="failed to get container status \"78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e\": rpc error: code = NotFound desc = could not find container \"78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e\": container with ID starting with 78a5a6d40583350cb695133285a6c905341173101248c60379478bd476a3578e not found: ID does not exist"
Sep 09 22:05:24.116182 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:24.116156    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"]
Sep 09 22:05:24.120320 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:24.120296    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/sub-test-model-3b2190-kserve-75bfd4bc49-pt9k9"]
Sep 09 22:05:25.074964 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.074930    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"]
Sep 09 22:05:25.075418 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.075357    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5faf582a-0e2a-4714-a55a-2741eee7cf32" containerName="main"
Sep 09 22:05:25.075418 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.075367    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5faf582a-0e2a-4714-a55a-2741eee7cf32" containerName="main"
Sep 09 22:05:25.075418 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.075380    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20920256-5d81-4b60-a218-05a32696d0bd" containerName="main"
Sep 09 22:05:25.075418 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.075386    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20920256-5d81-4b60-a218-05a32696d0bd" containerName="main"
Sep 09 22:05:25.075573 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.075452    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20920256-5d81-4b60-a218-05a32696d0bd" containerName="main"
Sep 09 22:05:25.075573 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.075461    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5faf582a-0e2a-4714-a55a-2741eee7cf32" containerName="main"
Sep 09 22:05:25.079739 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.079710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:05:25.079971 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.079952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.082705 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.082656    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ace629\"/\"openshift-service-ca.crt\""
Sep 09 22:05:25.084026 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.084000    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ace629\"/\"kube-root-ca.crt\""
Sep 09 22:05:25.084149 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.084063    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ace629\"/\"rate-test-model-ace629-kserve-self-signed-certs\""
Sep 09 22:05:25.084149 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.084107    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ace629\"/\"default-dockercfg-799s7\""
Sep 09 22:05:25.088946 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.088095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"]
Sep 09 22:05:25.141426 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.141396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-dshm\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.141606 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.141457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-model-cache\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.141606 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.141537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92j67\" (UniqueName: \"kubernetes.io/projected/35f33898-f4aa-4ba9-8e0f-988243c40f2d-kube-api-access-92j67\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.141747 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.141603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tls-certs\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.141747 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.141654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-home\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.141900 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.141760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tmp-dir\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.242900 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.242858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-model-cache\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.243108 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.242921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92j67\" (UniqueName: \"kubernetes.io/projected/35f33898-f4aa-4ba9-8e0f-988243c40f2d-kube-api-access-92j67\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.243108 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.242955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tls-certs\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.243108 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.242990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-home\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.243108 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.243045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tmp-dir\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.243108 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.243101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-dshm\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.243367 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.243314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-model-cache\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.243430 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.243366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-home\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.243430 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.243410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tmp-dir\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.245369 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.245348    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-dshm\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.245534 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.245516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tls-certs\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.251734 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.251715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92j67\" (UniqueName: \"kubernetes.io/projected/35f33898-f4aa-4ba9-8e0f-988243c40f2d-kube-api-access-92j67\") pod \"rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") " pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.392163 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.392073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:25.576316 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.576280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:25.607105 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.607074    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5faf582a-0e2a-4714-a55a-2741eee7cf32" path="/var/lib/kubelet/pods/5faf582a-0e2a-4714-a55a-2741eee7cf32/volumes"
Sep 09 22:05:25.671421 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:25.671363    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"]
Sep 09 22:05:25.673820 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:05:25.673788    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f33898_f4aa_4ba9_8e0f_988243c40f2d.slice/crio-05def2e05d1d8b08dd6adff86809b56c25858b68ec1e1cbf78edaa9618c4ce60 WatchSource:0}: Error finding container 05def2e05d1d8b08dd6adff86809b56c25858b68ec1e1cbf78edaa9618c4ce60: Status 404 returned error can't find the container with id 05def2e05d1d8b08dd6adff86809b56c25858b68ec1e1cbf78edaa9618c4ce60
Sep 09 22:05:26.114696 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:26.114513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z" event={"ID":"35f33898-f4aa-4ba9-8e0f-988243c40f2d","Type":"ContainerStarted","Data":"611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994"}
Sep 09 22:05:26.115093 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:26.114708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z" event={"ID":"35f33898-f4aa-4ba9-8e0f-988243c40f2d","Type":"ContainerStarted","Data":"05def2e05d1d8b08dd6adff86809b56c25858b68ec1e1cbf78edaa9618c4ce60"}
Sep 09 22:05:26.133313 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:26.133268    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z" podStartSLOduration=1.133253713 podStartE2EDuration="1.133253713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:26.13167675 +0000 UTC m=+1619.028342759" watchObservedRunningTime="2026-09-09 22:05:26.133253713 +0000 UTC m=+1619.029919721"
Sep 09 22:05:35.392325 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:35.392289    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:35.392977 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:35.392347    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:35.404970 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:35.404932    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:36.184435 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:36.184405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:05:41.094710 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:41.094674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:41.550185 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:41.550149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:41.869834 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:41.869721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:42.674475 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:42.674440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:44.957634 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:44.957597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:45.044435 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:45.044404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:46.041109 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.041074    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"]
Sep 09 22:05:46.046915 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.046890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.050587 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.050565    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4f06aa-serving-cert\""
Sep 09 22:05:46.060405 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.060369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"]
Sep 09 22:05:46.088399 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.088370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:46.139787 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.139734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-maas-api-tls\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.139977 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.139793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-metrics-certs\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.139977 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.139861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pflsf\" (UniqueName: \"kubernetes.io/projected/16b4201e-986f-4109-a111-dd7b4ba5ea84-kube-api-access-pflsf\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.139977 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.139884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b4201e-986f-4109-a111-dd7b4ba5ea84-openshift-service-ca\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.186225 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.186192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:46.241178 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.241143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-maas-api-tls\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.241178 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.241182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-metrics-certs\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.241405 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:05:46.241281    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-serving-cert: secret "maas-api-e2e-api-b-4f06aa-serving-cert" not found
Sep 09 22:05:46.241405 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.241312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pflsf\" (UniqueName: \"kubernetes.io/projected/16b4201e-986f-4109-a111-dd7b4ba5ea84-kube-api-access-pflsf\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.241405 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:05:46.241356    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-maas-api-tls podName:16b4201e-986f-4109-a111-dd7b4ba5ea84 nodeName:}" failed. No retries permitted until 2026-09-09 22:05:46.741335504 +0000 UTC m=+1639.638001501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-maas-api-tls") pod "maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p" (UID: "16b4201e-986f-4109-a111-dd7b4ba5ea84") : secret "maas-api-e2e-api-b-4f06aa-serving-cert" not found
Sep 09 22:05:46.241405 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.241379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b4201e-986f-4109-a111-dd7b4ba5ea84-openshift-service-ca\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.242051 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.242033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b4201e-986f-4109-a111-dd7b4ba5ea84-openshift-service-ca\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.243740 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.243710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-metrics-certs\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.253287 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.253256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pflsf\" (UniqueName: \"kubernetes.io/projected/16b4201e-986f-4109-a111-dd7b4ba5ea84-kube-api-access-pflsf\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.348988 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.348894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:46.633741 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.633660    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:46.730602 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.730565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:46.746819 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.746755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-maas-api-tls\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.749345 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.749320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-maas-api-tls\") pod \"maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:46.801299 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.801263    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"]
Sep 09 22:05:46.806002 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.805981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:05:46.808465 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.808443    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4f06aa\""
Sep 09 22:05:46.808553 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.808443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4f06aa-dockercfg-lk49s\""
Sep 09 22:05:46.815327 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.815299    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"]
Sep 09 22:05:46.848275 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.848241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n9jm\" (UniqueName: \"kubernetes.io/projected/789c8652-ee87-4186-9913-fa88013017af-kube-api-access-5n9jm\") pod \"payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl\" (UID: \"789c8652-ee87-4186-9913-fa88013017af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:05:46.848411 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.848294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789c8652-ee87-4186-9913-fa88013017af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl\" (UID: \"789c8652-ee87-4186-9913-fa88013017af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:05:46.948905 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.948822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n9jm\" (UniqueName: \"kubernetes.io/projected/789c8652-ee87-4186-9913-fa88013017af-kube-api-access-5n9jm\") pod \"payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl\" (UID: \"789c8652-ee87-4186-9913-fa88013017af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:05:46.948905 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.948867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789c8652-ee87-4186-9913-fa88013017af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl\" (UID: \"789c8652-ee87-4186-9913-fa88013017af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:05:46.949661 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.949639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789c8652-ee87-4186-9913-fa88013017af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl\" (UID: \"789c8652-ee87-4186-9913-fa88013017af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:05:46.957378 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.957350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n9jm\" (UniqueName: \"kubernetes.io/projected/789c8652-ee87-4186-9913-fa88013017af-kube-api-access-5n9jm\") pod \"payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl\" (UID: \"789c8652-ee87-4186-9913-fa88013017af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:05:46.960204 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:46.960184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:47.118489 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:47.118454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:05:47.209757 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:47.209668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:47.246576 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:47.246542    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"]
Sep 09 22:05:47.248349 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:05:47.248325    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b4201e_986f_4109_a111_dd7b4ba5ea84.slice/crio-e8f32678b1e56d5ebb990875903aeae39595e0b3d7f7279d6cfa0216cae12c39 WatchSource:0}: Error finding container e8f32678b1e56d5ebb990875903aeae39595e0b3d7f7279d6cfa0216cae12c39: Status 404 returned error can't find the container with id e8f32678b1e56d5ebb990875903aeae39595e0b3d7f7279d6cfa0216cae12c39
Sep 09 22:05:47.403635 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:47.398460    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"]
Sep 09 22:05:48.004511 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:48.004482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:48.244883 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:48.244841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p" event={"ID":"16b4201e-986f-4109-a111-dd7b4ba5ea84","Type":"ContainerStarted","Data":"35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7"}
Sep 09 22:05:48.245333 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:48.244908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p" event={"ID":"16b4201e-986f-4109-a111-dd7b4ba5ea84","Type":"ContainerStarted","Data":"e8f32678b1e56d5ebb990875903aeae39595e0b3d7f7279d6cfa0216cae12c39"}
Sep 09 22:05:48.245333 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:48.244941    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:48.246512 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:48.246480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl" event={"ID":"789c8652-ee87-4186-9913-fa88013017af","Type":"ContainerStarted","Data":"79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443"}
Sep 09 22:05:48.246625 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:48.246535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl" event={"ID":"789c8652-ee87-4186-9913-fa88013017af","Type":"ContainerStarted","Data":"293f26e346c4dfd3820f07e6e47280b098bc953b9411a154bee4d18e5487f940"}
Sep 09 22:05:48.246625 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:48.246582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:05:48.265391 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:48.265289    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p" podStartSLOduration=1.979478212 podStartE2EDuration="2.26527319s" totalImagesPullingTime="285.794978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:48.262918295 +0000 UTC m=+1641.159584304" watchObservedRunningTime="2026-09-09 22:05:48.26527319 +0000 UTC m=+1641.161939232"
Sep 09 22:05:48.283555 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:48.283503    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl" podStartSLOduration=2.283484075 podStartE2EDuration="2.283484075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:48.281356469 +0000 UTC m=+1641.178022498" watchObservedRunningTime="2026-09-09 22:05:48.283484075 +0000 UTC m=+1641.180150084"
Sep 09 22:05:54.258157 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:54.258129    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:05:59.256001 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:05:59.255965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:06:01.457719 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.457683    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"]
Sep 09 22:06:01.461278 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.461255    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.463810 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.463751    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"openshift-service-ca.crt\""
Sep 09 22:06:01.465282 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.465139    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"default-dockercfg-pkxqr\""
Sep 09 22:06:01.465282 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.465156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"kube-root-ca.crt\""
Sep 09 22:06:01.465282 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.465170    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b2190\"/\"rate-test-model-3b2190-kserve-self-signed-certs\""
Sep 09 22:06:01.470001 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.469980    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"]
Sep 09 22:06:01.484483 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.484462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:01.589349 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.589304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-home\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.589534 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.589362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tmp-dir\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.589534 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.589438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-model-cache\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.589534 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.589500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-dshm\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.589655 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.589563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tls-certs\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.589655 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.589596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m847k\" (UniqueName: \"kubernetes.io/projected/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-kube-api-access-m847k\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.690327 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.690288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-dshm\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.690497 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.690345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tls-certs\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.690497 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.690374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m847k\" (UniqueName: \"kubernetes.io/projected/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-kube-api-access-m847k\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.690497 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.690422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-home\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.690497 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.690467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tmp-dir\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.690731 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.690512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-model-cache\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.690858 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.690834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-home\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.690938 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.690905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tmp-dir\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.691014 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.690992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-model-cache\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.692810 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.692785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-dshm\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.693123 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.693107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tls-certs\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.700384 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.700365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m847k\" (UniqueName: \"kubernetes.io/projected/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-kube-api-access-m847k\") pod \"rate-test-model-3b2190-kserve-597cc59795-82tzz\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") " pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:01.771466 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:01.771425    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:02.055447 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:02.055417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"]
Sep 09 22:06:02.057205 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:06:02.057176    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a0b6e7b_2664_49dd_ae66_0efb2a161c93.slice/crio-874b3f8539d00294d2ec595f4c9fbf2d3ffda04a57e4027fca583941c6e4951d WatchSource:0}: Error finding container 874b3f8539d00294d2ec595f4c9fbf2d3ffda04a57e4027fca583941c6e4951d: Status 404 returned error can't find the container with id 874b3f8539d00294d2ec595f4c9fbf2d3ffda04a57e4027fca583941c6e4951d
Sep 09 22:06:02.335530 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:02.335434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz" event={"ID":"7a0b6e7b-2664-49dd-ae66-0efb2a161c93","Type":"ContainerStarted","Data":"80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8"}
Sep 09 22:06:02.335530 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:02.335478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz" event={"ID":"7a0b6e7b-2664-49dd-ae66-0efb2a161c93","Type":"ContainerStarted","Data":"874b3f8539d00294d2ec595f4c9fbf2d3ffda04a57e4027fca583941c6e4951d"}
Sep 09 22:06:02.353568 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:02.353519    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz" podStartSLOduration=1.353502775 podStartE2EDuration="1.353502775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:06:02.352403223 +0000 UTC m=+1655.249069244" watchObservedRunningTime="2026-09-09 22:06:02.353502775 +0000 UTC m=+1655.250168816"
Sep 09 22:06:09.955074 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:09.955036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:11.772389 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:11.772335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:11.772932 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:11.772408    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:11.785314 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:11.785273    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:12.410203 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:12.410172    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:06:13.084798 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:13.084740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:17.212150 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:17.212119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:22.982581 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:22.982548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:28.534731 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.534699    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"]
Sep 09 22:06:28.535272 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.534926    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl" podUID="a4b34d7f-f861-4066-bef8-df9514e9afce" containerName="maas-api" containerID="cri-o://e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e" gracePeriod=30
Sep 09 22:06:28.563133 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.563059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:28.635347 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.635313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:28.709169 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.709123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:28.791081 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.791018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:28.791520 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.791504    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:06:28.863090 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.863054    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l6fm\" (UniqueName: \"kubernetes.io/projected/a4b34d7f-f861-4066-bef8-df9514e9afce-kube-api-access-6l6fm\") pod \"a4b34d7f-f861-4066-bef8-df9514e9afce\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") "
Sep 09 22:06:28.863090 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.863094    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b34d7f-f861-4066-bef8-df9514e9afce-openshift-service-ca\") pod \"a4b34d7f-f861-4066-bef8-df9514e9afce\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") "
Sep 09 22:06:28.863290 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.863112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-maas-api-tls\") pod \"a4b34d7f-f861-4066-bef8-df9514e9afce\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") "
Sep 09 22:06:28.863290 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.863185    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-metrics-certs\") pod \"a4b34d7f-f861-4066-bef8-df9514e9afce\" (UID: \"a4b34d7f-f861-4066-bef8-df9514e9afce\") "
Sep 09 22:06:28.863517 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.863488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b34d7f-f861-4066-bef8-df9514e9afce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4b34d7f-f861-4066-bef8-df9514e9afce" (UID: "a4b34d7f-f861-4066-bef8-df9514e9afce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:06:28.865410 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.865380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4b34d7f-f861-4066-bef8-df9514e9afce" (UID: "a4b34d7f-f861-4066-bef8-df9514e9afce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:06:28.865513 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.865418    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b34d7f-f861-4066-bef8-df9514e9afce-kube-api-access-6l6fm" (OuterVolumeSpecName: "kube-api-access-6l6fm") pod "a4b34d7f-f861-4066-bef8-df9514e9afce" (UID: "a4b34d7f-f861-4066-bef8-df9514e9afce"). InnerVolumeSpecName "kube-api-access-6l6fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:06:28.865513 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.865503    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4b34d7f-f861-4066-bef8-df9514e9afce" (UID: "a4b34d7f-f861-4066-bef8-df9514e9afce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:06:28.964136 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.964091    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l6fm\" (UniqueName: \"kubernetes.io/projected/a4b34d7f-f861-4066-bef8-df9514e9afce-kube-api-access-6l6fm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:06:28.964136 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.964125    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b34d7f-f861-4066-bef8-df9514e9afce-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:06:28.964136 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.964135    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:06:28.964136 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:28.964143    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b34d7f-f861-4066-bef8-df9514e9afce-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:06:29.124163 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.124077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:29.505285 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.505252    2563 generic.go:356] "Generic (PLEG): container finished" podID="a4b34d7f-f861-4066-bef8-df9514e9afce" containerID="e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e" exitCode=0
Sep 09 22:06:29.505466 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.505297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl" event={"ID":"a4b34d7f-f861-4066-bef8-df9514e9afce","Type":"ContainerDied","Data":"e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e"}
Sep 09 22:06:29.505466 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.505318    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"
Sep 09 22:06:29.505466 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.505364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl" event={"ID":"a4b34d7f-f861-4066-bef8-df9514e9afce","Type":"ContainerDied","Data":"b3cac655ccd8429d2978d1d922f03774a9706e4a2a2e25c5590de8909d745e2b"}
Sep 09 22:06:29.505466 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.505387    2563 scope.go:117] "RemoveContainer" containerID="e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e"
Sep 09 22:06:29.515820 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.515801    2563 scope.go:117] "RemoveContainer" containerID="e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e"
Sep 09 22:06:29.516110 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:06:29.516090    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e\": container with ID starting with e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e not found: ID does not exist" containerID="e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e"
Sep 09 22:06:29.516172 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.516121    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e"} err="failed to get container status \"e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e\": rpc error: code = NotFound desc = could not find container \"e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e\": container with ID starting with e4c66bc64959cc110916b2ac1e68280fb5bd69281f03ccf20aec07821ce3ad9e not found: ID does not exist"
Sep 09 22:06:29.546677 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.546643    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"]
Sep 09 22:06:29.558123 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.558095    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a52f0-5584798775-h55pl"]
Sep 09 22:06:29.607291 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:29.607261    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4b34d7f-f861-4066-bef8-df9514e9afce" path="/var/lib/kubelet/pods/a4b34d7f-f861-4066-bef8-df9514e9afce/volumes"
Sep 09 22:06:34.887009 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:34.886977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:34.954881 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:34.954845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:35.141550 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:35.141470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:35.670092 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:35.670057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:51.073822 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.073723    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"]
Sep 09 22:06:51.074381 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.073998    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p" podUID="16b4201e-986f-4109-a111-dd7b4ba5ea84" containerName="maas-api" containerID="cri-o://35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7" gracePeriod=30
Sep 09 22:06:51.104618 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.104578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:51.135274 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.135238    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"]
Sep 09 22:06:51.135491 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.135467    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl" podUID="789c8652-ee87-4186-9913-fa88013017af" containerName="payload-pre-processing" containerID="cri-o://79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443" gracePeriod=30
Sep 09 22:06:51.177489 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.177433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:51.278470 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.278440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:51.341255 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.341174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:51.387119 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.387095    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:06:51.415158 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.415131    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:06:51.471942 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.471904    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pflsf\" (UniqueName: \"kubernetes.io/projected/16b4201e-986f-4109-a111-dd7b4ba5ea84-kube-api-access-pflsf\") pod \"16b4201e-986f-4109-a111-dd7b4ba5ea84\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") "
Sep 09 22:06:51.472141 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.471962    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n9jm\" (UniqueName: \"kubernetes.io/projected/789c8652-ee87-4186-9913-fa88013017af-kube-api-access-5n9jm\") pod \"789c8652-ee87-4186-9913-fa88013017af\" (UID: \"789c8652-ee87-4186-9913-fa88013017af\") "
Sep 09 22:06:51.472141 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.471996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-maas-api-tls\") pod \"16b4201e-986f-4109-a111-dd7b4ba5ea84\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") "
Sep 09 22:06:51.472141 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.472027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-metrics-certs\") pod \"16b4201e-986f-4109-a111-dd7b4ba5ea84\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") "
Sep 09 22:06:51.472141 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.472066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789c8652-ee87-4186-9913-fa88013017af-plugins-config-volume\") pod \"789c8652-ee87-4186-9913-fa88013017af\" (UID: \"789c8652-ee87-4186-9913-fa88013017af\") "
Sep 09 22:06:51.472382 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.472258    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b4201e-986f-4109-a111-dd7b4ba5ea84-openshift-service-ca\") pod \"16b4201e-986f-4109-a111-dd7b4ba5ea84\" (UID: \"16b4201e-986f-4109-a111-dd7b4ba5ea84\") "
Sep 09 22:06:51.472575 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.472456    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789c8652-ee87-4186-9913-fa88013017af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "789c8652-ee87-4186-9913-fa88013017af" (UID: "789c8652-ee87-4186-9913-fa88013017af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:06:51.472851 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.472667    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789c8652-ee87-4186-9913-fa88013017af-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:06:51.472919 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.472757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b4201e-986f-4109-a111-dd7b4ba5ea84-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16b4201e-986f-4109-a111-dd7b4ba5ea84" (UID: "16b4201e-986f-4109-a111-dd7b4ba5ea84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:06:51.474405 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.474375    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b4201e-986f-4109-a111-dd7b4ba5ea84-kube-api-access-pflsf" (OuterVolumeSpecName: "kube-api-access-pflsf") pod "16b4201e-986f-4109-a111-dd7b4ba5ea84" (UID: "16b4201e-986f-4109-a111-dd7b4ba5ea84"). InnerVolumeSpecName "kube-api-access-pflsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:06:51.474405 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.474386    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "16b4201e-986f-4109-a111-dd7b4ba5ea84" (UID: "16b4201e-986f-4109-a111-dd7b4ba5ea84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:06:51.474405 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.474388    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789c8652-ee87-4186-9913-fa88013017af-kube-api-access-5n9jm" (OuterVolumeSpecName: "kube-api-access-5n9jm") pod "789c8652-ee87-4186-9913-fa88013017af" (UID: "789c8652-ee87-4186-9913-fa88013017af"). InnerVolumeSpecName "kube-api-access-5n9jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:06:51.474754 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.474444    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16b4201e-986f-4109-a111-dd7b4ba5ea84" (UID: "16b4201e-986f-4109-a111-dd7b4ba5ea84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:06:51.573696 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.573653    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pflsf\" (UniqueName: \"kubernetes.io/projected/16b4201e-986f-4109-a111-dd7b4ba5ea84-kube-api-access-pflsf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:06:51.573696 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.573687    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n9jm\" (UniqueName: \"kubernetes.io/projected/789c8652-ee87-4186-9913-fa88013017af-kube-api-access-5n9jm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:06:51.573696 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.573701    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:06:51.573977 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.573712    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16b4201e-986f-4109-a111-dd7b4ba5ea84-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:06:51.573977 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.573723    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b4201e-986f-4109-a111-dd7b4ba5ea84-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:06:51.643881 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.643849    2563 generic.go:356] "Generic (PLEG): container finished" podID="16b4201e-986f-4109-a111-dd7b4ba5ea84" containerID="35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7" exitCode=0
Sep 09 22:06:51.644047 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.643901    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"
Sep 09 22:06:51.644047 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.643934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p" event={"ID":"16b4201e-986f-4109-a111-dd7b4ba5ea84","Type":"ContainerDied","Data":"35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7"}
Sep 09 22:06:51.644047 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.643978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p" event={"ID":"16b4201e-986f-4109-a111-dd7b4ba5ea84","Type":"ContainerDied","Data":"e8f32678b1e56d5ebb990875903aeae39595e0b3d7f7279d6cfa0216cae12c39"}
Sep 09 22:06:51.644047 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.644001    2563 scope.go:117] "RemoveContainer" containerID="35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7"
Sep 09 22:06:51.645428 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.645406    2563 generic.go:356] "Generic (PLEG): container finished" podID="789c8652-ee87-4186-9913-fa88013017af" containerID="79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443" exitCode=0
Sep 09 22:06:51.645557 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.645452    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"
Sep 09 22:06:51.645557 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.645502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl" event={"ID":"789c8652-ee87-4186-9913-fa88013017af","Type":"ContainerDied","Data":"79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443"}
Sep 09 22:06:51.645838 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.645721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl" event={"ID":"789c8652-ee87-4186-9913-fa88013017af","Type":"ContainerDied","Data":"293f26e346c4dfd3820f07e6e47280b098bc953b9411a154bee4d18e5487f940"}
Sep 09 22:06:51.653557 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.653539    2563 scope.go:117] "RemoveContainer" containerID="35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7"
Sep 09 22:06:51.653853 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:06:51.653833    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7\": container with ID starting with 35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7 not found: ID does not exist" containerID="35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7"
Sep 09 22:06:51.653923 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.653860    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7"} err="failed to get container status \"35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7\": rpc error: code = NotFound desc = could not find container \"35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7\": container with ID starting with 35064d4857d719b82705c3ac8faa641734203a58e86d60dc74346fee27651ab7 not found: ID does not exist"
Sep 09 22:06:51.653923 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.653876    2563 scope.go:117] "RemoveContainer" containerID="79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443"
Sep 09 22:06:51.656469 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.656453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:51.662885 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.662848    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"]
Sep 09 22:06:51.662953 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.662936    2563 scope.go:117] "RemoveContainer" containerID="79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443"
Sep 09 22:06:51.663225 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:06:51.663208    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443\": container with ID starting with 79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443 not found: ID does not exist" containerID="79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443"
Sep 09 22:06:51.663284 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.663231    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443"} err="failed to get container status \"79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443\": rpc error: code = NotFound desc = could not find container \"79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443\": container with ID starting with 79f22f3adced641f405072579ea443f38fa8a29a595add0eec2bb28fd6389443 not found: ID does not exist"
Sep 09 22:06:51.667420 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.667396    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f06aa-667f958d7d-45mfl"]
Sep 09 22:06:51.680974 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.680947    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"]
Sep 09 22:06:51.685667 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:51.685645    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f06aa-76b9bf4fc4-j8n2p"]
Sep 09 22:06:53.606749 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:53.606714    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16b4201e-986f-4109-a111-dd7b4ba5ea84" path="/var/lib/kubelet/pods/16b4201e-986f-4109-a111-dd7b4ba5ea84/volumes"
Sep 09 22:06:53.607130 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:53.607095    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="789c8652-ee87-4186-9913-fa88013017af" path="/var/lib/kubelet/pods/789c8652-ee87-4186-9913-fa88013017af/volumes"
Sep 09 22:06:56.479746 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:56.479708    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"]
Sep 09 22:06:56.494115 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:56.494072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:57.667325 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:57.667289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:57.683853 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:57.683824    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"]
Sep 09 22:06:57.765035 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:57.765002    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:57.922888 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:57.922804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:58.223333 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:58.223250    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:58.410146 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:58.410107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:58.486000 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:58.485914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:58.755265 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:06:58.755230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:01.482280 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:01.482245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:01.880119 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:01.880084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:01.973449 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:01.973410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:02.985385 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.985347    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"]
Sep 09 22:07:02.986021 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.985999    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b34d7f-f861-4066-bef8-df9514e9afce" containerName="maas-api"
Sep 09 22:07:02.986116 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.986023    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b34d7f-f861-4066-bef8-df9514e9afce" containerName="maas-api"
Sep 09 22:07:02.986116 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.986047    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789c8652-ee87-4186-9913-fa88013017af" containerName="payload-pre-processing"
Sep 09 22:07:02.986116 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.986053    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="789c8652-ee87-4186-9913-fa88013017af" containerName="payload-pre-processing"
Sep 09 22:07:02.986116 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.986063    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16b4201e-986f-4109-a111-dd7b4ba5ea84" containerName="maas-api"
Sep 09 22:07:02.986116 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.986070    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="16b4201e-986f-4109-a111-dd7b4ba5ea84" containerName="maas-api"
Sep 09 22:07:02.986394 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.986186    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="16b4201e-986f-4109-a111-dd7b4ba5ea84" containerName="maas-api"
Sep 09 22:07:02.986394 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.986197    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="789c8652-ee87-4186-9913-fa88013017af" containerName="payload-pre-processing"
Sep 09 22:07:02.986394 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.986206    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b34d7f-f861-4066-bef8-df9514e9afce" containerName="maas-api"
Sep 09 22:07:02.989901 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.989876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:02.992745 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:02.992723    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2f2a52-serving-cert\""
Sep 09 22:07:03.005888 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.005859    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"]
Sep 09 22:07:03.034762 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.034729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:03.087057 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.087028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.087226 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.087089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-metrics-certs\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.087226 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.087183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzgrp\" (UniqueName: \"kubernetes.io/projected/e963394f-de12-4685-b819-79c8c159d54f-kube-api-access-wzgrp\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.087305 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.087233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e963394f-de12-4685-b819-79c8c159d54f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.132868 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.132836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:03.188669 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.188631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-metrics-certs\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.188905 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.188689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzgrp\" (UniqueName: \"kubernetes.io/projected/e963394f-de12-4685-b819-79c8c159d54f-kube-api-access-wzgrp\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.188905 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.188728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e963394f-de12-4685-b819-79c8c159d54f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.188905 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.188809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.189678 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.189648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e963394f-de12-4685-b819-79c8c159d54f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.191476 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.191446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-metrics-certs\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.191566 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.191500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.197065 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.197039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzgrp\" (UniqueName: \"kubernetes.io/projected/e963394f-de12-4685-b819-79c8c159d54f-kube-api-access-wzgrp\") pod \"maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.232184 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.232149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:03.308357 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.308329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:03.505875 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.505830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:03.586813 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.586754    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"]
Sep 09 22:07:03.587427 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.587406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:03.589557 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:07:03.589530    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode963394f_de12_4685_b819_79c8c159d54f.slice/crio-bcfbe79807d77629dc2c501291b6a5fdc7e25673727762f313792702899e0e73 WatchSource:0}: Error finding container bcfbe79807d77629dc2c501291b6a5fdc7e25673727762f313792702899e0e73: Status 404 returned error can't find the container with id bcfbe79807d77629dc2c501291b6a5fdc7e25673727762f313792702899e0e73
Sep 09 22:07:03.653099 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.653058    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"]
Sep 09 22:07:03.658105 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.658081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:07:03.661004 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.660977    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2f2a52\""
Sep 09 22:07:03.661156 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.661040    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2f2a52-dockercfg-5hz7l\""
Sep 09 22:07:03.668346 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.668315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"]
Sep 09 22:07:03.725259 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.725226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz" event={"ID":"e963394f-de12-4685-b819-79c8c159d54f","Type":"ContainerStarted","Data":"bcfbe79807d77629dc2c501291b6a5fdc7e25673727762f313792702899e0e73"}
Sep 09 22:07:03.794891 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.794855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df17f921-9217-488c-a297-568e56c4e5a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg\" (UID: \"df17f921-9217-488c-a297-568e56c4e5a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:07:03.795133 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.795003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prmq8\" (UniqueName: \"kubernetes.io/projected/df17f921-9217-488c-a297-568e56c4e5a9-kube-api-access-prmq8\") pod \"payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg\" (UID: \"df17f921-9217-488c-a297-568e56c4e5a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:07:03.895855 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.895730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmq8\" (UniqueName: \"kubernetes.io/projected/df17f921-9217-488c-a297-568e56c4e5a9-kube-api-access-prmq8\") pod \"payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg\" (UID: \"df17f921-9217-488c-a297-568e56c4e5a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:07:03.896033 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.895864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df17f921-9217-488c-a297-568e56c4e5a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg\" (UID: \"df17f921-9217-488c-a297-568e56c4e5a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:07:03.896613 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.896591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df17f921-9217-488c-a297-568e56c4e5a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg\" (UID: \"df17f921-9217-488c-a297-568e56c4e5a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:07:03.906585 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.906558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmq8\" (UniqueName: \"kubernetes.io/projected/df17f921-9217-488c-a297-568e56c4e5a9-kube-api-access-prmq8\") pod \"payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg\" (UID: \"df17f921-9217-488c-a297-568e56c4e5a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:07:03.970570 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:03.970531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:07:04.264030 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:04.263997    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"]
Sep 09 22:07:04.264449 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:07:04.264417    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf17f921_9217_488c_a297_568e56c4e5a9.slice/crio-6bc60cc999a21ec70c26e8daefd8fe2ad3ec6d8c73b6f1f98a77b74af4312ac1 WatchSource:0}: Error finding container 6bc60cc999a21ec70c26e8daefd8fe2ad3ec6d8c73b6f1f98a77b74af4312ac1: Status 404 returned error can't find the container with id 6bc60cc999a21ec70c26e8daefd8fe2ad3ec6d8c73b6f1f98a77b74af4312ac1
Sep 09 22:07:04.734511 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:04.734468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg" event={"ID":"df17f921-9217-488c-a297-568e56c4e5a9","Type":"ContainerStarted","Data":"200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687"}
Sep 09 22:07:04.734723 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:04.734529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg" event={"ID":"df17f921-9217-488c-a297-568e56c4e5a9","Type":"ContainerStarted","Data":"6bc60cc999a21ec70c26e8daefd8fe2ad3ec6d8c73b6f1f98a77b74af4312ac1"}
Sep 09 22:07:04.734723 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:04.734560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:07:04.736435 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:04.736410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz" event={"ID":"e963394f-de12-4685-b819-79c8c159d54f","Type":"ContainerStarted","Data":"f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169"}
Sep 09 22:07:04.736555 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:04.736539    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:04.753007 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:04.752949    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg" podStartSLOduration=1.752934405 podStartE2EDuration="1.752934405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:07:04.750584877 +0000 UTC m=+1717.647250888" watchObservedRunningTime="2026-09-09 22:07:04.752934405 +0000 UTC m=+1717.649600404"
Sep 09 22:07:04.771434 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:04.771382    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz" podStartSLOduration=2.210732436 podStartE2EDuration="2.771365616s" totalImagesPullingTime="560.63318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:07:04.769618363 +0000 UTC m=+1717.666284370" watchObservedRunningTime="2026-09-09 22:07:04.771365616 +0000 UTC m=+1717.668031627"
Sep 09 22:07:08.426656 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:08.426621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:09.047657 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:09.047621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:09.125867 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:09.125822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:09.189065 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:09.189032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:09.266341 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:09.266309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:09.332003 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:09.331919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:09.633830 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:09.633719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:10.747878 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:10.747845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:07:11.501105 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.501066    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z" podUID="35f33898-f4aa-4ba9-8e0f-988243c40f2d" containerName="main" containerID="cri-o://611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994" gracePeriod=45
Sep 09 22:07:11.750015 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.749986    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:07:11.779194 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.779100    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tmp-dir\") pod \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") "
Sep 09 22:07:11.779194 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.779146    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-dshm\") pod \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") "
Sep 09 22:07:11.779194 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.779172    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tls-certs\") pod \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") "
Sep 09 22:07:11.779457 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.779215    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92j67\" (UniqueName: \"kubernetes.io/projected/35f33898-f4aa-4ba9-8e0f-988243c40f2d-kube-api-access-92j67\") pod \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") "
Sep 09 22:07:11.779457 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.779261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-model-cache\") pod \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") "
Sep 09 22:07:11.779457 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.779287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-home\") pod \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\" (UID: \"35f33898-f4aa-4ba9-8e0f-988243c40f2d\") "
Sep 09 22:07:11.779614 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.779472    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35f33898-f4aa-4ba9-8e0f-988243c40f2d" (UID: "35f33898-f4aa-4ba9-8e0f-988243c40f2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:11.779692 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.779662    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:11.779842 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.779703    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-home" (OuterVolumeSpecName: "home") pod "35f33898-f4aa-4ba9-8e0f-988243c40f2d" (UID: "35f33898-f4aa-4ba9-8e0f-988243c40f2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:11.779926 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.779898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "35f33898-f4aa-4ba9-8e0f-988243c40f2d" (UID: "35f33898-f4aa-4ba9-8e0f-988243c40f2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:11.782048 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.782004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35f33898-f4aa-4ba9-8e0f-988243c40f2d" (UID: "35f33898-f4aa-4ba9-8e0f-988243c40f2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:07:11.782161 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.782059    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-dshm" (OuterVolumeSpecName: "dshm") pod "35f33898-f4aa-4ba9-8e0f-988243c40f2d" (UID: "35f33898-f4aa-4ba9-8e0f-988243c40f2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:11.782161 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.782152    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f33898-f4aa-4ba9-8e0f-988243c40f2d-kube-api-access-92j67" (OuterVolumeSpecName: "kube-api-access-92j67") pod "35f33898-f4aa-4ba9-8e0f-988243c40f2d" (UID: "35f33898-f4aa-4ba9-8e0f-988243c40f2d"). InnerVolumeSpecName "kube-api-access-92j67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:11.784710 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.784685    2563 generic.go:356] "Generic (PLEG): container finished" podID="35f33898-f4aa-4ba9-8e0f-988243c40f2d" containerID="611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994" exitCode=0
Sep 09 22:07:11.784833 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.784740    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"
Sep 09 22:07:11.784833 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.784781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z" event={"ID":"35f33898-f4aa-4ba9-8e0f-988243c40f2d","Type":"ContainerDied","Data":"611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994"}
Sep 09 22:07:11.784833 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.784830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z" event={"ID":"35f33898-f4aa-4ba9-8e0f-988243c40f2d","Type":"ContainerDied","Data":"05def2e05d1d8b08dd6adff86809b56c25858b68ec1e1cbf78edaa9618c4ce60"}
Sep 09 22:07:11.784966 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.784852    2563 scope.go:117] "RemoveContainer" containerID="611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994"
Sep 09 22:07:11.809643 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.809622    2563 scope.go:117] "RemoveContainer" containerID="611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994"
Sep 09 22:07:11.810024 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:07:11.810000    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994\": container with ID starting with 611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994 not found: ID does not exist" containerID="611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994"
Sep 09 22:07:11.810133 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.810035    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994"} err="failed to get container status \"611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994\": rpc error: code = NotFound desc = could not find container \"611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994\": container with ID starting with 611944d18111a8b729a4a1921950304bfc21ff0163d28a99579b70a0aca7d994 not found: ID does not exist"
Sep 09 22:07:11.822990 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:07:11.822952    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f33898_f4aa_4ba9_8e0f_988243c40f2d.slice/crio-05def2e05d1d8b08dd6adff86809b56c25858b68ec1e1cbf78edaa9618c4ce60\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f33898_f4aa_4ba9_8e0f_988243c40f2d.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 22:07:11.823172 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:07:11.822967    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f33898_f4aa_4ba9_8e0f_988243c40f2d.slice/crio-05def2e05d1d8b08dd6adff86809b56c25858b68ec1e1cbf78edaa9618c4ce60\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f33898_f4aa_4ba9_8e0f_988243c40f2d.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 22:07:11.824274 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.824135    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"]
Sep 09 22:07:11.827488 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.827466    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ace629/rate-test-model-ace629-kserve-5f7d7cd7f-rfz6z"]
Sep 09 22:07:11.880616 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.880578    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f33898-f4aa-4ba9-8e0f-988243c40f2d-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:11.880616 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.880608    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92j67\" (UniqueName: \"kubernetes.io/projected/35f33898-f4aa-4ba9-8e0f-988243c40f2d-kube-api-access-92j67\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:11.880616 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.880619    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:11.880870 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.880629    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:11.880870 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:11.880636    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f33898-f4aa-4ba9-8e0f-988243c40f2d-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:12.704181 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.704134    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz" podUID="7a0b6e7b-2664-49dd-ae66-0efb2a161c93" containerName="main" containerID="cri-o://80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8" gracePeriod=45
Sep 09 22:07:12.825259 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.825219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:12.960141 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.960082    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:07:12.990597 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.990561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-model-cache\") pod \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") "
Sep 09 22:07:12.990823 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.990642    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m847k\" (UniqueName: \"kubernetes.io/projected/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-kube-api-access-m847k\") pod \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") "
Sep 09 22:07:12.990823 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.990676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-home\") pod \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") "
Sep 09 22:07:12.990823 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.990715    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-dshm\") pod \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") "
Sep 09 22:07:12.990823 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.990741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tls-certs\") pod \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") "
Sep 09 22:07:12.990823 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.990814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tmp-dir\") pod \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\" (UID: \"7a0b6e7b-2664-49dd-ae66-0efb2a161c93\") "
Sep 09 22:07:12.991105 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.990905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a0b6e7b-2664-49dd-ae66-0efb2a161c93" (UID: "7a0b6e7b-2664-49dd-ae66-0efb2a161c93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:12.991105 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.991004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-home" (OuterVolumeSpecName: "home") pod "7a0b6e7b-2664-49dd-ae66-0efb2a161c93" (UID: "7a0b6e7b-2664-49dd-ae66-0efb2a161c93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:12.991216 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.991137    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a0b6e7b-2664-49dd-ae66-0efb2a161c93" (UID: "7a0b6e7b-2664-49dd-ae66-0efb2a161c93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:12.991216 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.991197    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:12.991333 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.991216    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:12.993035 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.993010    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-kube-api-access-m847k" (OuterVolumeSpecName: "kube-api-access-m847k") pod "7a0b6e7b-2664-49dd-ae66-0efb2a161c93" (UID: "7a0b6e7b-2664-49dd-ae66-0efb2a161c93"). InnerVolumeSpecName "kube-api-access-m847k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:12.993134 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.993040    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-dshm" (OuterVolumeSpecName: "dshm") pod "7a0b6e7b-2664-49dd-ae66-0efb2a161c93" (UID: "7a0b6e7b-2664-49dd-ae66-0efb2a161c93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:12.993134 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:12.993099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a0b6e7b-2664-49dd-ae66-0efb2a161c93" (UID: "7a0b6e7b-2664-49dd-ae66-0efb2a161c93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:07:13.092554 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.092502    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:13.092554 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.092547    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:13.092554 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.092558    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:13.092554 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.092567    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m847k\" (UniqueName: \"kubernetes.io/projected/7a0b6e7b-2664-49dd-ae66-0efb2a161c93-kube-api-access-m847k\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:13.607142 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.607109    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35f33898-f4aa-4ba9-8e0f-988243c40f2d" path="/var/lib/kubelet/pods/35f33898-f4aa-4ba9-8e0f-988243c40f2d/volumes"
Sep 09 22:07:13.801144 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.801107    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a0b6e7b-2664-49dd-ae66-0efb2a161c93" containerID="80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8" exitCode=0
Sep 09 22:07:13.801316 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.801153    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"
Sep 09 22:07:13.801316 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.801193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz" event={"ID":"7a0b6e7b-2664-49dd-ae66-0efb2a161c93","Type":"ContainerDied","Data":"80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8"}
Sep 09 22:07:13.801316 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.801237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz" event={"ID":"7a0b6e7b-2664-49dd-ae66-0efb2a161c93","Type":"ContainerDied","Data":"874b3f8539d00294d2ec595f4c9fbf2d3ffda04a57e4027fca583941c6e4951d"}
Sep 09 22:07:13.801316 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.801258    2563 scope.go:117] "RemoveContainer" containerID="80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8"
Sep 09 22:07:13.814508 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.814489    2563 scope.go:117] "RemoveContainer" containerID="80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8"
Sep 09 22:07:13.814807 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:07:13.814785    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8\": container with ID starting with 80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8 not found: ID does not exist" containerID="80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8"
Sep 09 22:07:13.814861 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.814815    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8"} err="failed to get container status \"80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8\": rpc error: code = NotFound desc = could not find container \"80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8\": container with ID starting with 80e6e34494442c1f31993c2e28e3357f7393da2cb22cf4b0b740146f044c57e8 not found: ID does not exist"
Sep 09 22:07:13.819711 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.819688    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"]
Sep 09 22:07:13.822206 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:13.822187    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b2190/rate-test-model-3b2190-kserve-597cc59795-82tzz"]
Sep 09 22:07:15.435409 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:15.435366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:15.537016 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:15.536985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:15.606659 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:15.606625    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a0b6e7b-2664-49dd-ae66-0efb2a161c93" path="/var/lib/kubelet/pods/7a0b6e7b-2664-49dd-ae66-0efb2a161c93/volumes"
Sep 09 22:07:15.715279 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:15.715187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:15.744714 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:15.744672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:07:16.256635 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:16.256605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:25.031649 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:25.031617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:26.330010 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.329972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:26.368394 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.368361    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"]
Sep 09 22:07:26.368657 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.368599    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw" podUID="cb3074a9-5070-4f5e-af1c-c44482d414ea" containerName="payload-pre-processing" containerID="cri-o://61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e" gracePeriod=30
Sep 09 22:07:26.431271 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.431235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:26.531686 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.531655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:26.627602 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.627577    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:07:26.632062 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.632039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:26.728293 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.728260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3074a9-5070-4f5e-af1c-c44482d414ea-plugins-config-volume\") pod \"cb3074a9-5070-4f5e-af1c-c44482d414ea\" (UID: \"cb3074a9-5070-4f5e-af1c-c44482d414ea\") "
Sep 09 22:07:26.728468 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.728354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmhh6\" (UniqueName: \"kubernetes.io/projected/cb3074a9-5070-4f5e-af1c-c44482d414ea-kube-api-access-gmhh6\") pod \"cb3074a9-5070-4f5e-af1c-c44482d414ea\" (UID: \"cb3074a9-5070-4f5e-af1c-c44482d414ea\") "
Sep 09 22:07:26.728650 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.728627    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3074a9-5070-4f5e-af1c-c44482d414ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb3074a9-5070-4f5e-af1c-c44482d414ea" (UID: "cb3074a9-5070-4f5e-af1c-c44482d414ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:26.730573 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.730548    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3074a9-5070-4f5e-af1c-c44482d414ea-kube-api-access-gmhh6" (OuterVolumeSpecName: "kube-api-access-gmhh6") pod "cb3074a9-5070-4f5e-af1c-c44482d414ea" (UID: "cb3074a9-5070-4f5e-af1c-c44482d414ea"). InnerVolumeSpecName "kube-api-access-gmhh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:26.829930 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.829893    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmhh6\" (UniqueName: \"kubernetes.io/projected/cb3074a9-5070-4f5e-af1c-c44482d414ea-kube-api-access-gmhh6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.829930 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.829923    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3074a9-5070-4f5e-af1c-c44482d414ea-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.871356 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.871274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:26.881160 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.881132    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb3074a9-5070-4f5e-af1c-c44482d414ea" containerID="61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e" exitCode=0
Sep 09 22:07:26.881313 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.881171    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"
Sep 09 22:07:26.881313 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.881200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw" event={"ID":"cb3074a9-5070-4f5e-af1c-c44482d414ea","Type":"ContainerDied","Data":"61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e"}
Sep 09 22:07:26.881313 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.881245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw" event={"ID":"cb3074a9-5070-4f5e-af1c-c44482d414ea","Type":"ContainerDied","Data":"896e436e6bdb390a42b0ed9de79adab56d7c766c846b387c438107f440eca91a"}
Sep 09 22:07:26.881313 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.881270    2563 scope.go:117] "RemoveContainer" containerID="61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e"
Sep 09 22:07:26.891432 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.891410    2563 scope.go:117] "RemoveContainer" containerID="61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e"
Sep 09 22:07:26.891689 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:07:26.891671    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e\": container with ID starting with 61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e not found: ID does not exist" containerID="61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e"
Sep 09 22:07:26.891754 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.891698    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e"} err="failed to get container status \"61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e\": rpc error: code = NotFound desc = could not find container \"61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e\": container with ID starting with 61decbc1275d235a8d1c0265b52d7c887609a874da950ac86ba83e5e495ece7e not found: ID does not exist"
Sep 09 22:07:26.909699 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.909674    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"]
Sep 09 22:07:26.914019 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.913995    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e5cb43-c7f9b6fb6-5dzrw"]
Sep 09 22:07:26.973204 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:26.973178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:27.556302 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:27.556264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:27.607254 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:27.607225    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb3074a9-5070-4f5e-af1c-c44482d414ea" path="/var/lib/kubelet/pods/cb3074a9-5070-4f5e-af1c-c44482d414ea/volumes"
Sep 09 22:07:27.852738 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:27.852665    2563 scope.go:117] "RemoveContainer" containerID="ddaad823dc84a9eca17edec2a5fe357da7a36ca1b9896071fa5eb20215018221"
Sep 09 22:07:27.864014 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:27.863986    2563 scope.go:117] "RemoveContainer" containerID="ce767bfbb8b7a328ee83a230068e09dfc91d626a4cc1cd9f7aaf41c09a0b8787"
Sep 09 22:07:27.892977 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:27.892954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:28.243884 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:28.243860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:30.470413 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:30.470374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:30.654910 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:30.654876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:31.607741 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:31.607714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:31.706163 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:31.706133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:32.047928 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:32.047886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:32.200050 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:32.200020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:32.299127 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:32.299052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:33.676454 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:33.676419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:33.757992 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:33.757948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:33.987808 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:33.987700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:55.442655 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:07:55.442612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:00.521292 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:00.521264    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9pcwv_d5b4be3d-5973-4de5-ad82-0582fb44a438/payload-processing/0.log"
Sep 09 22:08:08.915232 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.915189    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"]
Sep 09 22:08:08.915864 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.915816    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0b6e7b-2664-49dd-ae66-0efb2a161c93" containerName="main"
Sep 09 22:08:08.915864 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.915836    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0b6e7b-2664-49dd-ae66-0efb2a161c93" containerName="main"
Sep 09 22:08:08.915864 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.915857    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f33898-f4aa-4ba9-8e0f-988243c40f2d" containerName="main"
Sep 09 22:08:08.915864 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.915865    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f33898-f4aa-4ba9-8e0f-988243c40f2d" containerName="main"
Sep 09 22:08:08.916097 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.915902    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3074a9-5070-4f5e-af1c-c44482d414ea" containerName="payload-pre-processing"
Sep 09 22:08:08.916097 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.915910    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3074a9-5070-4f5e-af1c-c44482d414ea" containerName="payload-pre-processing"
Sep 09 22:08:08.916097 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.916016    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f33898-f4aa-4ba9-8e0f-988243c40f2d" containerName="main"
Sep 09 22:08:08.916097 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.916030    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a0b6e7b-2664-49dd-ae66-0efb2a161c93" containerName="main"
Sep 09 22:08:08.916097 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.916041    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3074a9-5070-4f5e-af1c-c44482d414ea" containerName="payload-pre-processing"
Sep 09 22:08:08.919539 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.919518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:08.922373 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.922350    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2f2a52\"/\"openshift-service-ca.crt\""
Sep 09 22:08:08.923465 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.923437    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2f2a52\"/\"default-dockercfg-6rpgt\""
Sep 09 22:08:08.923561 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.923466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2f2a52\"/\"ipp-route-2f2a52-kserve-self-signed-certs\""
Sep 09 22:08:08.923561 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.923483    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2f2a52\"/\"kube-root-ca.crt\""
Sep 09 22:08:08.988289 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.988255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"]
Sep 09 22:08:08.997922 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:08.997898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:09.021188 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.021157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-dshm\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.021188 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.021192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-model-cache\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.021419 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.021301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-tmp-dir\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.021419 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.021351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25bd51e-b510-49fc-b288-380b6c066ee1-tls-certs\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.021419 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.021393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjhth\" (UniqueName: \"kubernetes.io/projected/c25bd51e-b510-49fc-b288-380b6c066ee1-kube-api-access-gjhth\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.021607 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.021436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-home\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.122869 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.122836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-tmp-dir\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.123052 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.122878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25bd51e-b510-49fc-b288-380b6c066ee1-tls-certs\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.123052 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.122912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjhth\" (UniqueName: \"kubernetes.io/projected/c25bd51e-b510-49fc-b288-380b6c066ee1-kube-api-access-gjhth\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.123052 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.122955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-home\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.123052 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.123009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-dshm\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.123052 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.123034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-model-cache\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.123330 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.123302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-tmp-dir\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.123422 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.123392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-home\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.123551 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.123489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-model-cache\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.125483 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.125458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-dshm\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.126146 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.126128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25bd51e-b510-49fc-b288-380b6c066ee1-tls-certs\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.132466 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.132447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjhth\" (UniqueName: \"kubernetes.io/projected/c25bd51e-b510-49fc-b288-380b6c066ee1-kube-api-access-gjhth\") pod \"ipp-route-2f2a52-kserve-587f6ddf79-jrcgs\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") " pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.228462 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.228377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:09.504252 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:09.504228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"]
Sep 09 22:08:09.506537 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:08:09.506508    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc25bd51e_b510_49fc_b288_380b6c066ee1.slice/crio-43c2be36195a83388babc415684d94748d341b23a9b8e27c2b57b68c2ee49c0f WatchSource:0}: Error finding container 43c2be36195a83388babc415684d94748d341b23a9b8e27c2b57b68c2ee49c0f: Status 404 returned error can't find the container with id 43c2be36195a83388babc415684d94748d341b23a9b8e27c2b57b68c2ee49c0f
Sep 09 22:08:10.137002 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:10.136966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs" event={"ID":"c25bd51e-b510-49fc-b288-380b6c066ee1","Type":"ContainerStarted","Data":"84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c"}
Sep 09 22:08:10.137371 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:10.137015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs" event={"ID":"c25bd51e-b510-49fc-b288-380b6c066ee1","Type":"ContainerStarted","Data":"43c2be36195a83388babc415684d94748d341b23a9b8e27c2b57b68c2ee49c0f"}
Sep 09 22:08:10.163453 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:10.163405    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs" podStartSLOduration=2.16338879 podStartE2EDuration="2.16338879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:08:10.161531183 +0000 UTC m=+1783.058197190" watchObservedRunningTime="2026-09-09 22:08:10.16338879 +0000 UTC m=+1783.060054798"
Sep 09 22:08:17.608492 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:17.608457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:17.673373 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:17.673345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:19.218258 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:19.218182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:19.229116 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:19.229084    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:19.229266 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:19.229131    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:19.241608 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:19.241573    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:20.217401 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:20.217350    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:08:27.699489 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:27.699455    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 22:08:27.709659 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:27.709638    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 22:08:32.504547 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:32.504509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:32.564781 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:32.564722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:36.329330 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:36.329298    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg_df17f921-9217-488c-a297-568e56c4e5a9/payload-processing/0.log"
Sep 09 22:08:36.446162 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:36.446129    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9pcwv_d5b4be3d-5973-4de5-ad82-0582fb44a438/payload-processing/0.log"
Sep 09 22:08:36.853498 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:36.853460    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:37.188339 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:37.188248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:37.401384 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:37.401348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:39.844250 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:39.844219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:39.932194 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:39.932163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:41.171204 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:41.171168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:41.272513 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:41.272468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:41.375050 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:41.375024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:41.665845 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:41.665817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:08:41.767919 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:08:41.767890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:04.814742 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:04.814704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:11.526192 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:11.526155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:11.628660 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:11.628630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:11.730089 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:11.730058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:11.828950 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:11.828859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:12.099640 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:12.099563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:17.895990 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:17.895953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:17.961930 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:17.961901    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:18.190428 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:18.190341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:21.915367 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:21.915333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:27.928623 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:27.928589    2563 scope.go:117] "RemoveContainer" containerID="26093a4d477182f126f0b983570d4700e9afac545bebf58add88b19821d0836b"
Sep 09 22:09:33.270675 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:33.270641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:37.254717 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.254679    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"]
Sep 09 22:09:37.255270 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.254961    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz" podUID="e963394f-de12-4685-b819-79c8c159d54f" containerName="maas-api" containerID="cri-o://f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169" gracePeriod=30
Sep 09 22:09:37.287717 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.287669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:37.316347 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.316313    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"]
Sep 09 22:09:37.316580 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.316558    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg" podUID="df17f921-9217-488c-a297-568e56c4e5a9" containerName="payload-processing" containerID="cri-o://200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687" gracePeriod=30
Sep 09 22:09:37.357955 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.357643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:37.458313 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.458275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:37.516805 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.516683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:37.580703 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.580681    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:09:37.603149 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.603121    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:09:37.663440 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.663406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e963394f-de12-4685-b819-79c8c159d54f-openshift-service-ca\") pod \"e963394f-de12-4685-b819-79c8c159d54f\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") "
Sep 09 22:09:37.663627 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.663495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-maas-api-tls\") pod \"e963394f-de12-4685-b819-79c8c159d54f\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") "
Sep 09 22:09:37.663701 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.663654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzgrp\" (UniqueName: \"kubernetes.io/projected/e963394f-de12-4685-b819-79c8c159d54f-kube-api-access-wzgrp\") pod \"e963394f-de12-4685-b819-79c8c159d54f\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") "
Sep 09 22:09:37.663805 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.663756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-metrics-certs\") pod \"e963394f-de12-4685-b819-79c8c159d54f\" (UID: \"e963394f-de12-4685-b819-79c8c159d54f\") "
Sep 09 22:09:37.663920 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.663897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e963394f-de12-4685-b819-79c8c159d54f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e963394f-de12-4685-b819-79c8c159d54f" (UID: "e963394f-de12-4685-b819-79c8c159d54f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:09:37.664122 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.664101    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e963394f-de12-4685-b819-79c8c159d54f-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:37.665905 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.665877    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e963394f-de12-4685-b819-79c8c159d54f" (UID: "e963394f-de12-4685-b819-79c8c159d54f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:09:37.666112 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.666090    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e963394f-de12-4685-b819-79c8c159d54f-kube-api-access-wzgrp" (OuterVolumeSpecName: "kube-api-access-wzgrp") pod "e963394f-de12-4685-b819-79c8c159d54f" (UID: "e963394f-de12-4685-b819-79c8c159d54f"). InnerVolumeSpecName "kube-api-access-wzgrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:09:37.666209 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.666107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e963394f-de12-4685-b819-79c8c159d54f" (UID: "e963394f-de12-4685-b819-79c8c159d54f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:09:37.668438 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.668414    2563 generic.go:356] "Generic (PLEG): container finished" podID="df17f921-9217-488c-a297-568e56c4e5a9" containerID="200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687" exitCode=0
Sep 09 22:09:37.668531 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.668460    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"
Sep 09 22:09:37.668531 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.668503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg" event={"ID":"df17f921-9217-488c-a297-568e56c4e5a9","Type":"ContainerDied","Data":"200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687"}
Sep 09 22:09:37.668647 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.668570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg" event={"ID":"df17f921-9217-488c-a297-568e56c4e5a9","Type":"ContainerDied","Data":"6bc60cc999a21ec70c26e8daefd8fe2ad3ec6d8c73b6f1f98a77b74af4312ac1"}
Sep 09 22:09:37.668647 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.668603    2563 scope.go:117] "RemoveContainer" containerID="200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687"
Sep 09 22:09:37.669924 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.669902    2563 generic.go:356] "Generic (PLEG): container finished" podID="e963394f-de12-4685-b819-79c8c159d54f" containerID="f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169" exitCode=0
Sep 09 22:09:37.670014 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.669929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz" event={"ID":"e963394f-de12-4685-b819-79c8c159d54f","Type":"ContainerDied","Data":"f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169"}
Sep 09 22:09:37.670014 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.669956    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"
Sep 09 22:09:37.670014 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.669979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz" event={"ID":"e963394f-de12-4685-b819-79c8c159d54f","Type":"ContainerDied","Data":"bcfbe79807d77629dc2c501291b6a5fdc7e25673727762f313792702899e0e73"}
Sep 09 22:09:37.678501 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.678485    2563 scope.go:117] "RemoveContainer" containerID="200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687"
Sep 09 22:09:37.678845 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:09:37.678819    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687\": container with ID starting with 200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687 not found: ID does not exist" containerID="200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687"
Sep 09 22:09:37.678957 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.678847    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687"} err="failed to get container status \"200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687\": rpc error: code = NotFound desc = could not find container \"200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687\": container with ID starting with 200c99fba4c50d6ee0048e65a6cf711db0ad328da5bd4e09ea1ed1e22b55a687 not found: ID does not exist"
Sep 09 22:09:37.678957 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.678862    2563 scope.go:117] "RemoveContainer" containerID="f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169"
Sep 09 22:09:37.687633 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.687616    2563 scope.go:117] "RemoveContainer" containerID="f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169"
Sep 09 22:09:37.687906 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:09:37.687886    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169\": container with ID starting with f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169 not found: ID does not exist" containerID="f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169"
Sep 09 22:09:37.687960 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.687913    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169"} err="failed to get container status \"f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169\": rpc error: code = NotFound desc = could not find container \"f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169\": container with ID starting with f9a821353c96183912392babc91eec1ea2261c0cd4173b3193c930d75e242169 not found: ID does not exist"
Sep 09 22:09:37.693317 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.693298    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"]
Sep 09 22:09:37.697200 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.697180    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2f2a52-5f98664cc6-9clgz"]
Sep 09 22:09:37.765257 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.765228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prmq8\" (UniqueName: \"kubernetes.io/projected/df17f921-9217-488c-a297-568e56c4e5a9-kube-api-access-prmq8\") pod \"df17f921-9217-488c-a297-568e56c4e5a9\" (UID: \"df17f921-9217-488c-a297-568e56c4e5a9\") "
Sep 09 22:09:37.765420 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.765328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df17f921-9217-488c-a297-568e56c4e5a9-plugins-config-volume\") pod \"df17f921-9217-488c-a297-568e56c4e5a9\" (UID: \"df17f921-9217-488c-a297-568e56c4e5a9\") "
Sep 09 22:09:37.765623 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.765600    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:37.765623 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.765619    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzgrp\" (UniqueName: \"kubernetes.io/projected/e963394f-de12-4685-b819-79c8c159d54f-kube-api-access-wzgrp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:37.765796 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.765633    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e963394f-de12-4685-b819-79c8c159d54f-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:37.765869 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.765759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df17f921-9217-488c-a297-568e56c4e5a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df17f921-9217-488c-a297-568e56c4e5a9" (UID: "df17f921-9217-488c-a297-568e56c4e5a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:09:37.767794 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.767728    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df17f921-9217-488c-a297-568e56c4e5a9-kube-api-access-prmq8" (OuterVolumeSpecName: "kube-api-access-prmq8") pod "df17f921-9217-488c-a297-568e56c4e5a9" (UID: "df17f921-9217-488c-a297-568e56c4e5a9"). InnerVolumeSpecName "kube-api-access-prmq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:09:37.866429 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.866391    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prmq8\" (UniqueName: \"kubernetes.io/projected/df17f921-9217-488c-a297-568e56c4e5a9-kube-api-access-prmq8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:37.866429 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.866427    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df17f921-9217-488c-a297-568e56c4e5a9-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:37.874608 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.874579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:37.990739 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.990706    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"]
Sep 09 22:09:37.995256 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:37.995233    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2f2a52-5b9c84dc9c-j8jdg"]
Sep 09 22:09:39.605862 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:39.605827    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df17f921-9217-488c-a297-568e56c4e5a9" path="/var/lib/kubelet/pods/df17f921-9217-488c-a297-568e56c4e5a9/volumes"
Sep 09 22:09:39.606313 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:39.606215    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e963394f-de12-4685-b819-79c8c159d54f" path="/var/lib/kubelet/pods/e963394f-de12-4685-b819-79c8c159d54f/volumes"
Sep 09 22:09:42.846395 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:42.846365    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"]
Sep 09 22:09:42.942460 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:42.942431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:44.594409 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:44.594380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:09:57.866734 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:57.866642    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs" podUID="c25bd51e-b510-49fc-b288-380b6c066ee1" containerName="main" containerID="cri-o://84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c" gracePeriod=45
Sep 09 22:09:58.122897 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.122838    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:09:58.256060 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256030    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25bd51e-b510-49fc-b288-380b6c066ee1-tls-certs\") pod \"c25bd51e-b510-49fc-b288-380b6c066ee1\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") "
Sep 09 22:09:58.256233 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256093    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-home\") pod \"c25bd51e-b510-49fc-b288-380b6c066ee1\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") "
Sep 09 22:09:58.256233 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjhth\" (UniqueName: \"kubernetes.io/projected/c25bd51e-b510-49fc-b288-380b6c066ee1-kube-api-access-gjhth\") pod \"c25bd51e-b510-49fc-b288-380b6c066ee1\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") "
Sep 09 22:09:58.256233 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256174    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-tmp-dir\") pod \"c25bd51e-b510-49fc-b288-380b6c066ee1\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") "
Sep 09 22:09:58.256233 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-model-cache\") pod \"c25bd51e-b510-49fc-b288-380b6c066ee1\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") "
Sep 09 22:09:58.256439 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256243    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-dshm\") pod \"c25bd51e-b510-49fc-b288-380b6c066ee1\" (UID: \"c25bd51e-b510-49fc-b288-380b6c066ee1\") "
Sep 09 22:09:58.256439 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256357    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-home" (OuterVolumeSpecName: "home") pod "c25bd51e-b510-49fc-b288-380b6c066ee1" (UID: "c25bd51e-b510-49fc-b288-380b6c066ee1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:09:58.256548 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256500    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c25bd51e-b510-49fc-b288-380b6c066ee1" (UID: "c25bd51e-b510-49fc-b288-380b6c066ee1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:09:58.256548 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256520    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-model-cache" (OuterVolumeSpecName: "model-cache") pod "c25bd51e-b510-49fc-b288-380b6c066ee1" (UID: "c25bd51e-b510-49fc-b288-380b6c066ee1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:09:58.256692 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256675    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:58.256748 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256698    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:58.256748 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.256712    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:58.258517 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.258486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-dshm" (OuterVolumeSpecName: "dshm") pod "c25bd51e-b510-49fc-b288-380b6c066ee1" (UID: "c25bd51e-b510-49fc-b288-380b6c066ee1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:09:58.258617 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.258530    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25bd51e-b510-49fc-b288-380b6c066ee1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c25bd51e-b510-49fc-b288-380b6c066ee1" (UID: "c25bd51e-b510-49fc-b288-380b6c066ee1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:09:58.258617 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.258588    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25bd51e-b510-49fc-b288-380b6c066ee1-kube-api-access-gjhth" (OuterVolumeSpecName: "kube-api-access-gjhth") pod "c25bd51e-b510-49fc-b288-380b6c066ee1" (UID: "c25bd51e-b510-49fc-b288-380b6c066ee1"). InnerVolumeSpecName "kube-api-access-gjhth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:09:58.357839 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.357809    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25bd51e-b510-49fc-b288-380b6c066ee1-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:58.357839 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.357837    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25bd51e-b510-49fc-b288-380b6c066ee1-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:58.358033 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.357849    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjhth\" (UniqueName: \"kubernetes.io/projected/c25bd51e-b510-49fc-b288-380b6c066ee1-kube-api-access-gjhth\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:09:58.794520 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.794485    2563 generic.go:356] "Generic (PLEG): container finished" podID="c25bd51e-b510-49fc-b288-380b6c066ee1" containerID="84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c" exitCode=0
Sep 09 22:09:58.794698 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.794534    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"
Sep 09 22:09:58.794698 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.794575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs" event={"ID":"c25bd51e-b510-49fc-b288-380b6c066ee1","Type":"ContainerDied","Data":"84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c"}
Sep 09 22:09:58.794698 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.794621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs" event={"ID":"c25bd51e-b510-49fc-b288-380b6c066ee1","Type":"ContainerDied","Data":"43c2be36195a83388babc415684d94748d341b23a9b8e27c2b57b68c2ee49c0f"}
Sep 09 22:09:58.794698 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.794637    2563 scope.go:117] "RemoveContainer" containerID="84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c"
Sep 09 22:09:58.804622 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.804606    2563 scope.go:117] "RemoveContainer" containerID="84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c"
Sep 09 22:09:58.804907 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:09:58.804889    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c\": container with ID starting with 84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c not found: ID does not exist" containerID="84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c"
Sep 09 22:09:58.804964 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.804916    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c"} err="failed to get container status \"84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c\": rpc error: code = NotFound desc = could not find container \"84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c\": container with ID starting with 84791ca09bf6161ef7ab481a69ed9c702ee2244be07d64d1e8d229a0447d839c not found: ID does not exist"
Sep 09 22:09:58.816220 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.816197    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"]
Sep 09 22:09:58.820272 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:58.820252    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2f2a52/ipp-route-2f2a52-kserve-587f6ddf79-jrcgs"]
Sep 09 22:09:59.606750 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:09:59.606714    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c25bd51e-b510-49fc-b288-380b6c066ee1" path="/var/lib/kubelet/pods/c25bd51e-b510-49fc-b288-380b6c066ee1/volumes"
Sep 09 22:10:13.677009 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:10:13.676962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:10:18.985071 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:10:18.985019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:10:20.302389 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:10:20.302355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:10:26.144730 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:10:26.144699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:10:49.066275 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:10:49.066235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:10:49.165276 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:10:49.165231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:10:49.335562 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:10:49.335462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:10:49.889118 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:10:49.889089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:00.178988 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:00.178951    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:00.244859 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:00.244827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:00.344866 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:00.344828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:00.418135 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:00.418108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:00.734850 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:00.734816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:06.341007 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:06.340968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:06.402904 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:06.402866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:06.656669 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:06.656589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:07.065149 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:07.065117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:17.462741 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.462705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:17.484925 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.484894    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"]
Sep 09 22:11:17.485110 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.485090    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj" podUID="423c2f38-05fc-494d-8468-9dd124dc7c59" containerName="payload-processing" containerID="cri-o://700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a" gracePeriod=30
Sep 09 22:11:17.533742 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.533702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:17.635469 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.635430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:17.735507 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.735481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:17.755344 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.755319    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 22:11:17.881940 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.881902    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/423c2f38-05fc-494d-8468-9dd124dc7c59-plugins-config-volume\") pod \"423c2f38-05fc-494d-8468-9dd124dc7c59\" (UID: \"423c2f38-05fc-494d-8468-9dd124dc7c59\") "
Sep 09 22:11:17.882101 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.882026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq69m\" (UniqueName: \"kubernetes.io/projected/423c2f38-05fc-494d-8468-9dd124dc7c59-kube-api-access-dq69m\") pod \"423c2f38-05fc-494d-8468-9dd124dc7c59\" (UID: \"423c2f38-05fc-494d-8468-9dd124dc7c59\") "
Sep 09 22:11:17.882273 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.882249    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423c2f38-05fc-494d-8468-9dd124dc7c59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "423c2f38-05fc-494d-8468-9dd124dc7c59" (UID: "423c2f38-05fc-494d-8468-9dd124dc7c59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:11:17.884234 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.884207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423c2f38-05fc-494d-8468-9dd124dc7c59-kube-api-access-dq69m" (OuterVolumeSpecName: "kube-api-access-dq69m") pod "423c2f38-05fc-494d-8468-9dd124dc7c59" (UID: "423c2f38-05fc-494d-8468-9dd124dc7c59"). InnerVolumeSpecName "kube-api-access-dq69m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:17.982787 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.982742    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/423c2f38-05fc-494d-8468-9dd124dc7c59-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:17.982787 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:17.982787    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq69m\" (UniqueName: \"kubernetes.io/projected/423c2f38-05fc-494d-8468-9dd124dc7c59-kube-api-access-dq69m\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:18.007376 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:18.007350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:18.253146 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:18.253112    2563 generic.go:356] "Generic (PLEG): container finished" podID="423c2f38-05fc-494d-8468-9dd124dc7c59" containerID="700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a" exitCode=0
Sep 09 22:11:18.253146 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:18.253151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj" event={"ID":"423c2f38-05fc-494d-8468-9dd124dc7c59","Type":"ContainerDied","Data":"700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a"}
Sep 09 22:11:18.253415 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:18.253169    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"
Sep 09 22:11:18.253415 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:18.253194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj" event={"ID":"423c2f38-05fc-494d-8468-9dd124dc7c59","Type":"ContainerDied","Data":"eb7d1fd2bd1020499a0a76c7cc42a196df74c7b0a10419839e101a11e28eb85d"}
Sep 09 22:11:18.253415 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:18.253214    2563 scope.go:117] "RemoveContainer" containerID="700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a"
Sep 09 22:11:18.263233 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:18.263213    2563 scope.go:117] "RemoveContainer" containerID="700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a"
Sep 09 22:11:18.263491 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:11:18.263472    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a\": container with ID starting with 700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a not found: ID does not exist" containerID="700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a"
Sep 09 22:11:18.263555 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:18.263498    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a"} err="failed to get container status \"700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a\": rpc error: code = NotFound desc = could not find container \"700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a\": container with ID starting with 700e980c92d347a4b0a8801c0966e2e80cd11c8ce3fdbe001ef15ffb864c570a not found: ID does not exist"
Sep 09 22:11:18.276960 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:18.276935    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"]
Sep 09 22:11:18.282017 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:18.281995    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ace629-5859c7ffc9-zk6kj"]
Sep 09 22:11:19.606203 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:19.606170    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423c2f38-05fc-494d-8468-9dd124dc7c59" path="/var/lib/kubelet/pods/423c2f38-05fc-494d-8468-9dd124dc7c59/volumes"
Sep 09 22:11:23.884577 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:23.884543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:23.984412 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:23.984382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:24.231336 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:24.231249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:24.709580 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:24.709542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:27.975448 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:27.975413    2563 scope.go:117] "RemoveContainer" containerID="6c599f884ebca4c4a0fdb4ad87139f8981009229ef6efe0f2f4e844d147979f2"
Sep 09 22:11:34.938464 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:34.938432    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"]
Sep 09 22:11:34.938994 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:34.938670    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk" podUID="71470ad2-0883-4473-8ab3-8c5e963871d1" containerName="maas-api" containerID="cri-o://76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47" gracePeriod=30
Sep 09 22:11:34.970869 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:34.970841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:35.001035 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.001001    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"]
Sep 09 22:11:35.001365 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.001251    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z" podUID="2e96ca93-b449-445e-b2a8-0dd4cb76ee78" containerName="payload-pre-processing" containerID="cri-o://1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158" gracePeriod=30
Sep 09 22:11:35.070511 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.070467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:35.161876 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.161837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:35.217224 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.217182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:35.243671 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.243641    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 22:11:35.278702 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.278653    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 22:11:35.337456 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.337429    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-plugins-config-volume\") pod \"2e96ca93-b449-445e-b2a8-0dd4cb76ee78\" (UID: \"2e96ca93-b449-445e-b2a8-0dd4cb76ee78\") "
Sep 09 22:11:35.337691 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.337501    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr924\" (UniqueName: \"kubernetes.io/projected/71470ad2-0883-4473-8ab3-8c5e963871d1-kube-api-access-qr924\") pod \"71470ad2-0883-4473-8ab3-8c5e963871d1\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") "
Sep 09 22:11:35.337691 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.337532    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-metrics-certs\") pod \"71470ad2-0883-4473-8ab3-8c5e963871d1\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") "
Sep 09 22:11:35.337691 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.337548    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x946s\" (UniqueName: \"kubernetes.io/projected/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-kube-api-access-x946s\") pod \"2e96ca93-b449-445e-b2a8-0dd4cb76ee78\" (UID: \"2e96ca93-b449-445e-b2a8-0dd4cb76ee78\") "
Sep 09 22:11:35.337691 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.337592    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71470ad2-0883-4473-8ab3-8c5e963871d1-openshift-service-ca\") pod \"71470ad2-0883-4473-8ab3-8c5e963871d1\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") "
Sep 09 22:11:35.337691 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.337626    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-maas-api-tls\") pod \"71470ad2-0883-4473-8ab3-8c5e963871d1\" (UID: \"71470ad2-0883-4473-8ab3-8c5e963871d1\") "
Sep 09 22:11:35.337994 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.337923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e96ca93-b449-445e-b2a8-0dd4cb76ee78" (UID: "2e96ca93-b449-445e-b2a8-0dd4cb76ee78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:11:35.338068 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.338036    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71470ad2-0883-4473-8ab3-8c5e963871d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71470ad2-0883-4473-8ab3-8c5e963871d1" (UID: "71470ad2-0883-4473-8ab3-8c5e963871d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:11:35.340057 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.340017    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71470ad2-0883-4473-8ab3-8c5e963871d1" (UID: "71470ad2-0883-4473-8ab3-8c5e963871d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:11:35.340171 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.340058    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-kube-api-access-x946s" (OuterVolumeSpecName: "kube-api-access-x946s") pod "2e96ca93-b449-445e-b2a8-0dd4cb76ee78" (UID: "2e96ca93-b449-445e-b2a8-0dd4cb76ee78"). InnerVolumeSpecName "kube-api-access-x946s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:35.340171 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.340097    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71470ad2-0883-4473-8ab3-8c5e963871d1" (UID: "71470ad2-0883-4473-8ab3-8c5e963871d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:11:35.340171 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.340095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71470ad2-0883-4473-8ab3-8c5e963871d1-kube-api-access-qr924" (OuterVolumeSpecName: "kube-api-access-qr924") pod "71470ad2-0883-4473-8ab3-8c5e963871d1" (UID: "71470ad2-0883-4473-8ab3-8c5e963871d1"). InnerVolumeSpecName "kube-api-access-qr924". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:35.350242 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.350218    2563 generic.go:356] "Generic (PLEG): container finished" podID="2e96ca93-b449-445e-b2a8-0dd4cb76ee78" containerID="1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158" exitCode=0
Sep 09 22:11:35.350349 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.350259    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"
Sep 09 22:11:35.350349 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.350286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z" event={"ID":"2e96ca93-b449-445e-b2a8-0dd4cb76ee78","Type":"ContainerDied","Data":"1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158"}
Sep 09 22:11:35.350349 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.350338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z" event={"ID":"2e96ca93-b449-445e-b2a8-0dd4cb76ee78","Type":"ContainerDied","Data":"5bf65be4a27a0778ce52f7d0a952ed00418b3e372442efb21c7b2cd6b153e5ef"}
Sep 09 22:11:35.350500 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.350363    2563 scope.go:117] "RemoveContainer" containerID="1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158"
Sep 09 22:11:35.351729 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.351706    2563 generic.go:356] "Generic (PLEG): container finished" podID="71470ad2-0883-4473-8ab3-8c5e963871d1" containerID="76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47" exitCode=0
Sep 09 22:11:35.352010 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.351742    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"
Sep 09 22:11:35.352010 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.351807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk" event={"ID":"71470ad2-0883-4473-8ab3-8c5e963871d1","Type":"ContainerDied","Data":"76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47"}
Sep 09 22:11:35.352010 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.351863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk" event={"ID":"71470ad2-0883-4473-8ab3-8c5e963871d1","Type":"ContainerDied","Data":"48aba10dd2e151eed6c255cd0be61fb5237e6b32fd29146f6fced44c09f6bc98"}
Sep 09 22:11:35.359503 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.359485    2563 scope.go:117] "RemoveContainer" containerID="1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158"
Sep 09 22:11:35.359826 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:11:35.359801    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158\": container with ID starting with 1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158 not found: ID does not exist" containerID="1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158"
Sep 09 22:11:35.359916 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.359839    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158"} err="failed to get container status \"1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158\": rpc error: code = NotFound desc = could not find container \"1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158\": container with ID starting with 1e182a74cf6e77ea5b215111f592e9396f2589722381de2c4f6f52c343bb0158 not found: ID does not exist"
Sep 09 22:11:35.359916 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.359864    2563 scope.go:117] "RemoveContainer" containerID="76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47"
Sep 09 22:11:35.368521 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.368506    2563 scope.go:117] "RemoveContainer" containerID="76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47"
Sep 09 22:11:35.368784 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:11:35.368752    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47\": container with ID starting with 76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47 not found: ID does not exist" containerID="76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47"
Sep 09 22:11:35.368842 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.368789    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47"} err="failed to get container status \"76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47\": rpc error: code = NotFound desc = could not find container \"76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47\": container with ID starting with 76fd4c9a7b0deb36645ad2e3c597a77a0b500cc471c1cf2f176d1b182b931c47 not found: ID does not exist"
Sep 09 22:11:35.376906 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.376885    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"]
Sep 09 22:11:35.380670 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.380649    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3b2190-5dd99dbc55-9ztsk"]
Sep 09 22:11:35.391216 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.391195    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"]
Sep 09 22:11:35.394107 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.394087    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b2190-645c48f8cd-7pf4z"]
Sep 09 22:11:35.438486 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.438457    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:35.438486 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.438483    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x946s\" (UniqueName: \"kubernetes.io/projected/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-kube-api-access-x946s\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:35.438639 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.438498    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71470ad2-0883-4473-8ab3-8c5e963871d1-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:35.438639 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.438514    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71470ad2-0883-4473-8ab3-8c5e963871d1-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:35.438639 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.438527    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96ca93-b449-445e-b2a8-0dd4cb76ee78-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:35.438639 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.438541    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr924\" (UniqueName: \"kubernetes.io/projected/71470ad2-0883-4473-8ab3-8c5e963871d1-kube-api-access-qr924\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:35.544380 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.544353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:35.606408 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.606370    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e96ca93-b449-445e-b2a8-0dd4cb76ee78" path="/var/lib/kubelet/pods/2e96ca93-b449-445e-b2a8-0dd4cb76ee78/volumes"
Sep 09 22:11:35.606877 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:35.606857    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71470ad2-0883-4473-8ab3-8c5e963871d1" path="/var/lib/kubelet/pods/71470ad2-0883-4473-8ab3-8c5e963871d1/volumes"
Sep 09 22:11:41.665294 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:41.665256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:41.737920 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:41.737887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:41.929179 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:41.929089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:11:44.331373 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.331332    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-kj7nh"]
Sep 09 22:11:44.331894 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.331590    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c999c7bf-kj7nh" podUID="0cb4fc34-ed19-4e4f-8523-978122c6fa9e" containerName="manager" containerID="cri-o://6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb" gracePeriod=30
Sep 09 22:11:44.570227 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.570203    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 22:11:44.622546 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.622456    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-webhook-cert\") pod \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") "
Sep 09 22:11:44.622546 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.622501    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfdjn\" (UniqueName: \"kubernetes.io/projected/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-kube-api-access-vfdjn\") pod \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") "
Sep 09 22:11:44.622546 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.622529    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-metrics-certs\") pod \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\" (UID: \"0cb4fc34-ed19-4e4f-8523-978122c6fa9e\") "
Sep 09 22:11:44.624794 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.624728    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0cb4fc34-ed19-4e4f-8523-978122c6fa9e" (UID: "0cb4fc34-ed19-4e4f-8523-978122c6fa9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:11:44.624931 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.624840    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0cb4fc34-ed19-4e4f-8523-978122c6fa9e" (UID: "0cb4fc34-ed19-4e4f-8523-978122c6fa9e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:11:44.624931 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.624914    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-kube-api-access-vfdjn" (OuterVolumeSpecName: "kube-api-access-vfdjn") pod "0cb4fc34-ed19-4e4f-8523-978122c6fa9e" (UID: "0cb4fc34-ed19-4e4f-8523-978122c6fa9e"). InnerVolumeSpecName "kube-api-access-vfdjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:44.723553 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.723509    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-webhook-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:44.723553 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.723545    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfdjn\" (UniqueName: \"kubernetes.io/projected/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-kube-api-access-vfdjn\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:44.723553 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:44.723554    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb4fc34-ed19-4e4f-8523-978122c6fa9e-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:11:45.411512 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:45.411480    2563 generic.go:356] "Generic (PLEG): container finished" podID="0cb4fc34-ed19-4e4f-8523-978122c6fa9e" containerID="6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb" exitCode=0
Sep 09 22:11:45.411959 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:45.411530    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c999c7bf-kj7nh"
Sep 09 22:11:45.411959 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:45.411567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c999c7bf-kj7nh" event={"ID":"0cb4fc34-ed19-4e4f-8523-978122c6fa9e","Type":"ContainerDied","Data":"6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb"}
Sep 09 22:11:45.411959 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:45.411614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c999c7bf-kj7nh" event={"ID":"0cb4fc34-ed19-4e4f-8523-978122c6fa9e","Type":"ContainerDied","Data":"225a27d04a33c9d5fd086c1f937cd6192137764df3eeadfdd10809c2098a94c9"}
Sep 09 22:11:45.411959 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:45.411637    2563 scope.go:117] "RemoveContainer" containerID="6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb"
Sep 09 22:11:45.422218 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:45.422200    2563 scope.go:117] "RemoveContainer" containerID="6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb"
Sep 09 22:11:45.422488 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:11:45.422470    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb\": container with ID starting with 6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb not found: ID does not exist" containerID="6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb"
Sep 09 22:11:45.422556 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:45.422501    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb"} err="failed to get container status \"6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb\": rpc error: code = NotFound desc = could not find container \"6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb\": container with ID starting with 6dab590439d48a6193ca49a5de03e0e36dacf4baef3a45a31f665e6e8fb0d4fb not found: ID does not exist"
Sep 09 22:11:45.433980 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:45.433949    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-kj7nh"]
Sep 09 22:11:45.436855 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:45.436835    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-kj7nh"]
Sep 09 22:11:45.605864 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:45.605829    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb4fc34-ed19-4e4f-8523-978122c6fa9e" path="/var/lib/kubelet/pods/0cb4fc34-ed19-4e4f-8523-978122c6fa9e/volumes"
Sep 09 22:11:57.964905 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:11:57.964870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:12:30.171994 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:30.171953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:12:30.272417 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:30.272386    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:12:46.207445 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207403    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c999c7bf-vmbzf"]
Sep 09 22:12:46.207884 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207868    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71470ad2-0883-4473-8ab3-8c5e963871d1" containerName="maas-api"
Sep 09 22:12:46.207939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207887    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71470ad2-0883-4473-8ab3-8c5e963871d1" containerName="maas-api"
Sep 09 22:12:46.207939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207901    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423c2f38-05fc-494d-8468-9dd124dc7c59" containerName="payload-processing"
Sep 09 22:12:46.207939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207907    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="423c2f38-05fc-494d-8468-9dd124dc7c59" containerName="payload-processing"
Sep 09 22:12:46.207939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207915    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e96ca93-b449-445e-b2a8-0dd4cb76ee78" containerName="payload-pre-processing"
Sep 09 22:12:46.207939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207921    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e96ca93-b449-445e-b2a8-0dd4cb76ee78" containerName="payload-pre-processing"
Sep 09 22:12:46.207939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207932    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb4fc34-ed19-4e4f-8523-978122c6fa9e" containerName="manager"
Sep 09 22:12:46.207939 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207940    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb4fc34-ed19-4e4f-8523-978122c6fa9e" containerName="manager"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207955    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df17f921-9217-488c-a297-568e56c4e5a9" containerName="payload-processing"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207960    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="df17f921-9217-488c-a297-568e56c4e5a9" containerName="payload-processing"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207968    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e963394f-de12-4685-b819-79c8c159d54f" containerName="maas-api"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207974    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e963394f-de12-4685-b819-79c8c159d54f" containerName="maas-api"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207980    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c25bd51e-b510-49fc-b288-380b6c066ee1" containerName="main"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.207985    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c25bd51e-b510-49fc-b288-380b6c066ee1" containerName="main"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.208058    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71470ad2-0883-4473-8ab3-8c5e963871d1" containerName="maas-api"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.208066    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e96ca93-b449-445e-b2a8-0dd4cb76ee78" containerName="payload-pre-processing"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.208072    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c25bd51e-b510-49fc-b288-380b6c066ee1" containerName="main"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.208079    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb4fc34-ed19-4e4f-8523-978122c6fa9e" containerName="manager"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.208085    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="df17f921-9217-488c-a297-568e56c4e5a9" containerName="payload-processing"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.208090    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="423c2f38-05fc-494d-8468-9dd124dc7c59" containerName="payload-processing"
Sep 09 22:12:46.208154 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.208097    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e963394f-de12-4685-b819-79c8c159d54f" containerName="maas-api"
Sep 09 22:12:46.211165 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.211150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.215295 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.215274    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 22:12:46.215295 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.215287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 22:12:46.215495 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.215275    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8wcbp\""
Sep 09 22:12:46.215495 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.215314    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 22:12:46.221479 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.221455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-vmbzf"]
Sep 09 22:12:46.366016 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.365958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsdm\" (UniqueName: \"kubernetes.io/projected/f2cb06cc-6e00-40c5-aef8-757b96b4645a-kube-api-access-zmsdm\") pod \"maas-controller-5c999c7bf-vmbzf\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") " pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.366241 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.366150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-webhook-cert\") pod \"maas-controller-5c999c7bf-vmbzf\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") " pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.366241 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.366192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-metrics-certs\") pod \"maas-controller-5c999c7bf-vmbzf\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") " pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.467534 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.467428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsdm\" (UniqueName: \"kubernetes.io/projected/f2cb06cc-6e00-40c5-aef8-757b96b4645a-kube-api-access-zmsdm\") pod \"maas-controller-5c999c7bf-vmbzf\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") " pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.467534 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.467529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-webhook-cert\") pod \"maas-controller-5c999c7bf-vmbzf\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") " pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.467534 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.467548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-metrics-certs\") pod \"maas-controller-5c999c7bf-vmbzf\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") " pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.470193 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.470168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-webhook-cert\") pod \"maas-controller-5c999c7bf-vmbzf\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") " pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.470295 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.470268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-metrics-certs\") pod \"maas-controller-5c999c7bf-vmbzf\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") " pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.476725 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.476700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsdm\" (UniqueName: \"kubernetes.io/projected/f2cb06cc-6e00-40c5-aef8-757b96b4645a-kube-api-access-zmsdm\") pod \"maas-controller-5c999c7bf-vmbzf\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") " pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.521723 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.521691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:46.796259 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.796235    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-vmbzf"]
Sep 09 22:12:46.798605 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:12:46.798571    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cb06cc_6e00_40c5_aef8_757b96b4645a.slice/crio-11aaed1294398f5ad4570c60d4eb1be30c3261246d9ba710f36715140f0bbcf2 WatchSource:0}: Error finding container 11aaed1294398f5ad4570c60d4eb1be30c3261246d9ba710f36715140f0bbcf2: Status 404 returned error can't find the container with id 11aaed1294398f5ad4570c60d4eb1be30c3261246d9ba710f36715140f0bbcf2
Sep 09 22:12:46.803249 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:46.803226    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 22:12:47.756231 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:47.756196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c999c7bf-vmbzf" event={"ID":"f2cb06cc-6e00-40c5-aef8-757b96b4645a","Type":"ContainerStarted","Data":"e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f"}
Sep 09 22:12:47.756597 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:47.756245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c999c7bf-vmbzf" event={"ID":"f2cb06cc-6e00-40c5-aef8-757b96b4645a","Type":"ContainerStarted","Data":"11aaed1294398f5ad4570c60d4eb1be30c3261246d9ba710f36715140f0bbcf2"}
Sep 09 22:12:47.756597 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:47.756277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:12:47.778124 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:47.778080    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c999c7bf-vmbzf" podStartSLOduration=1.2780273979999999 podStartE2EDuration="1.778064798s" totalImagesPullingTime="500.0374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:12:47.7753335 +0000 UTC m=+2060.671999570" watchObservedRunningTime="2026-09-09 22:12:47.778064798 +0000 UTC m=+2060.674730805"
Sep 09 22:12:58.764329 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:12:58.764247    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:13:08.318171 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:08.318129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:13:10.105783 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.105729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:13:10.209752 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.209714    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-vmbzf"]
Sep 09 22:13:10.210012 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.209966    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c999c7bf-vmbzf" podUID="f2cb06cc-6e00-40c5-aef8-757b96b4645a" containerName="manager" containerID="cri-o://e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f" gracePeriod=30
Sep 09 22:13:10.450726 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.450702    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:13:10.476179 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.476145    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-metrics-certs\") pod \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") "
Sep 09 22:13:10.476362 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.476247    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-webhook-cert\") pod \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") "
Sep 09 22:13:10.476362 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.476287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmsdm\" (UniqueName: \"kubernetes.io/projected/f2cb06cc-6e00-40c5-aef8-757b96b4645a-kube-api-access-zmsdm\") pod \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\" (UID: \"f2cb06cc-6e00-40c5-aef8-757b96b4645a\") "
Sep 09 22:13:10.479196 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.479164    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cb06cc-6e00-40c5-aef8-757b96b4645a-kube-api-access-zmsdm" (OuterVolumeSpecName: "kube-api-access-zmsdm") pod "f2cb06cc-6e00-40c5-aef8-757b96b4645a" (UID: "f2cb06cc-6e00-40c5-aef8-757b96b4645a"). InnerVolumeSpecName "kube-api-access-zmsdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:13:10.479479 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.479441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f2cb06cc-6e00-40c5-aef8-757b96b4645a" (UID: "f2cb06cc-6e00-40c5-aef8-757b96b4645a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:13:10.479479 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.479455    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f2cb06cc-6e00-40c5-aef8-757b96b4645a" (UID: "f2cb06cc-6e00-40c5-aef8-757b96b4645a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:13:10.577351 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.577321    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmsdm\" (UniqueName: \"kubernetes.io/projected/f2cb06cc-6e00-40c5-aef8-757b96b4645a-kube-api-access-zmsdm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:13:10.577351 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.577348    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:13:10.577351 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.577359    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2cb06cc-6e00-40c5-aef8-757b96b4645a-webhook-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:13:10.883747 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.883713    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2cb06cc-6e00-40c5-aef8-757b96b4645a" containerID="e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f" exitCode=0
Sep 09 22:13:10.883957 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.883761    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c999c7bf-vmbzf"
Sep 09 22:13:10.883957 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.883801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c999c7bf-vmbzf" event={"ID":"f2cb06cc-6e00-40c5-aef8-757b96b4645a","Type":"ContainerDied","Data":"e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f"}
Sep 09 22:13:10.883957 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.883862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c999c7bf-vmbzf" event={"ID":"f2cb06cc-6e00-40c5-aef8-757b96b4645a","Type":"ContainerDied","Data":"11aaed1294398f5ad4570c60d4eb1be30c3261246d9ba710f36715140f0bbcf2"}
Sep 09 22:13:10.883957 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.883890    2563 scope.go:117] "RemoveContainer" containerID="e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f"
Sep 09 22:13:10.893484 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.893467    2563 scope.go:117] "RemoveContainer" containerID="e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f"
Sep 09 22:13:10.893762 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:13:10.893744    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f\": container with ID starting with e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f not found: ID does not exist" containerID="e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f"
Sep 09 22:13:10.893826 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.893792    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f"} err="failed to get container status \"e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f\": rpc error: code = NotFound desc = could not find container \"e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f\": container with ID starting with e1e7ba5fbeb7248af9ea0d4eb54344d0d87679083eef23a44f5dd0328e1e3d0f not found: ID does not exist"
Sep 09 22:13:10.906163 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.906138    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-vmbzf"]
Sep 09 22:13:10.910298 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:10.910276    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-vmbzf"]
Sep 09 22:13:11.575748 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.575716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:13:11.605932 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.605905    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cb06cc-6e00-40c5-aef8-757b96b4645a" path="/var/lib/kubelet/pods/f2cb06cc-6e00-40c5-aef8-757b96b4645a/volumes"
Sep 09 22:13:11.695331 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.695298    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c999c7bf-9jr7t"]
Sep 09 22:13:11.695674 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.695661    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cb06cc-6e00-40c5-aef8-757b96b4645a" containerName="manager"
Sep 09 22:13:11.695718 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.695676    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cb06cc-6e00-40c5-aef8-757b96b4645a" containerName="manager"
Sep 09 22:13:11.695778 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.695756    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cb06cc-6e00-40c5-aef8-757b96b4645a" containerName="manager"
Sep 09 22:13:11.700174 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.700156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:11.703059 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.703041    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 22:13:11.703180 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.703046    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8wcbp\""
Sep 09 22:13:11.704198 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.704171    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 22:13:11.704286 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.704213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 22:13:11.710460 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.710438    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-9jr7t"]
Sep 09 22:13:11.789495 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.789464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7rv\" (UniqueName: \"kubernetes.io/projected/6c9d113a-6c44-4e06-8d67-210126253147-kube-api-access-mw7rv\") pod \"maas-controller-5c999c7bf-9jr7t\" (UID: \"6c9d113a-6c44-4e06-8d67-210126253147\") " pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:11.789678 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.789531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c9d113a-6c44-4e06-8d67-210126253147-webhook-cert\") pod \"maas-controller-5c999c7bf-9jr7t\" (UID: \"6c9d113a-6c44-4e06-8d67-210126253147\") " pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:11.789678 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.789582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9d113a-6c44-4e06-8d67-210126253147-metrics-certs\") pod \"maas-controller-5c999c7bf-9jr7t\" (UID: \"6c9d113a-6c44-4e06-8d67-210126253147\") " pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:11.890066 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.889988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7rv\" (UniqueName: \"kubernetes.io/projected/6c9d113a-6c44-4e06-8d67-210126253147-kube-api-access-mw7rv\") pod \"maas-controller-5c999c7bf-9jr7t\" (UID: \"6c9d113a-6c44-4e06-8d67-210126253147\") " pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:11.890066 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.890049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c9d113a-6c44-4e06-8d67-210126253147-webhook-cert\") pod \"maas-controller-5c999c7bf-9jr7t\" (UID: \"6c9d113a-6c44-4e06-8d67-210126253147\") " pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:11.890262 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.890083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9d113a-6c44-4e06-8d67-210126253147-metrics-certs\") pod \"maas-controller-5c999c7bf-9jr7t\" (UID: \"6c9d113a-6c44-4e06-8d67-210126253147\") " pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:11.892723 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.892690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9d113a-6c44-4e06-8d67-210126253147-metrics-certs\") pod \"maas-controller-5c999c7bf-9jr7t\" (UID: \"6c9d113a-6c44-4e06-8d67-210126253147\") " pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:11.892864 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.892690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c9d113a-6c44-4e06-8d67-210126253147-webhook-cert\") pod \"maas-controller-5c999c7bf-9jr7t\" (UID: \"6c9d113a-6c44-4e06-8d67-210126253147\") " pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:11.898371 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:11.898345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7rv\" (UniqueName: \"kubernetes.io/projected/6c9d113a-6c44-4e06-8d67-210126253147-kube-api-access-mw7rv\") pod \"maas-controller-5c999c7bf-9jr7t\" (UID: \"6c9d113a-6c44-4e06-8d67-210126253147\") " pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:12.011729 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:12.011694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:12.289112 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:12.289076    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c999c7bf-9jr7t"]
Sep 09 22:13:12.290664 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:13:12.290631    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9d113a_6c44_4e06_8d67_210126253147.slice/crio-334756cd3bde496578a53c56862756b6a1d99ad7e23093d1dd8d085722d52796 WatchSource:0}: Error finding container 334756cd3bde496578a53c56862756b6a1d99ad7e23093d1dd8d085722d52796: Status 404 returned error can't find the container with id 334756cd3bde496578a53c56862756b6a1d99ad7e23093d1dd8d085722d52796
Sep 09 22:13:12.898263 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:12.898170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c999c7bf-9jr7t" event={"ID":"6c9d113a-6c44-4e06-8d67-210126253147","Type":"ContainerStarted","Data":"c597e5d40545980aae129f35eb8a1bfe620de3d510ca23886eb0cbede990c21a"}
Sep 09 22:13:12.898263 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:12.898244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c999c7bf-9jr7t" event={"ID":"6c9d113a-6c44-4e06-8d67-210126253147","Type":"ContainerStarted","Data":"334756cd3bde496578a53c56862756b6a1d99ad7e23093d1dd8d085722d52796"}
Sep 09 22:13:12.898762 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:12.898359    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:12.919388 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:12.919336    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c999c7bf-9jr7t" podStartSLOduration=1.6158884900000001 podStartE2EDuration="1.919320521s" totalImagesPullingTime="303.432031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:13:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:13:12.918622286 +0000 UTC m=+2085.815288296" watchObservedRunningTime="2026-09-09 22:13:12.919320521 +0000 UTC m=+2085.815986529"
Sep 09 22:13:23.906531 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:23.906498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c999c7bf-9jr7t"
Sep 09 22:13:27.737820 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:27.737793    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 22:13:27.748698 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:13:27.748672    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 22:16:21.429450 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:21.429412    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:16:25.972493 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:25.972462    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"]
Sep 09 22:16:25.976114 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:25.976098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:25.978744 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:25.978724    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-93d910aa-kserve-self-signed-certs\""
Sep 09 22:16:25.984298 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:25.984274    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"]
Sep 09 22:16:26.056408 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.056375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:16:26.114019 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.113988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tls-certs\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.114187 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.114051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-home\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.114187 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.114076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-dshm\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.114187 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.114109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-model-cache\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.114187 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.114132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tmp-dir\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.114325 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.114204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6zrk\" (UniqueName: \"kubernetes.io/projected/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-kube-api-access-s6zrk\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.215630 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.215595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-home\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.215630 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.215632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-dshm\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.215945 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.215663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-model-cache\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.215945 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.215683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tmp-dir\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.215945 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.215746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6zrk\" (UniqueName: \"kubernetes.io/projected/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-kube-api-access-s6zrk\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.215945 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.215809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tls-certs\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.216172 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.216074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-home\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.216172 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.216133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-model-cache\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.216263 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.216242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tmp-dir\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.218231 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.218204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-dshm\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.218451 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.218434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tls-certs\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.223965 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.223914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6zrk\" (UniqueName: \"kubernetes.io/projected/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-kube-api-access-s6zrk\") pod \"e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") " pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.285503 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.285462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:26.562838 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.562812    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"]
Sep 09 22:16:26.563856 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:16:26.563824    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa460355_e76f_4ebb_ad5a_55e3fdc61a69.slice/crio-f813d554978aff536939e80ca2b2e5b22f4bed320dd114ef0578ebd0d24dbecd WatchSource:0}: Error finding container f813d554978aff536939e80ca2b2e5b22f4bed320dd114ef0578ebd0d24dbecd: Status 404 returned error can't find the container with id f813d554978aff536939e80ca2b2e5b22f4bed320dd114ef0578ebd0d24dbecd
Sep 09 22:16:26.974451 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.974363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p" event={"ID":"aa460355-e76f-4ebb-ad5a-55e3fdc61a69","Type":"ContainerStarted","Data":"2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e"}
Sep 09 22:16:26.974451 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.974410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p" event={"ID":"aa460355-e76f-4ebb-ad5a-55e3fdc61a69","Type":"ContainerStarted","Data":"f813d554978aff536939e80ca2b2e5b22f4bed320dd114ef0578ebd0d24dbecd"}
Sep 09 22:16:26.994355 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:26.994305    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p" podStartSLOduration=1.994288598 podStartE2EDuration="1.994288598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:16:26.99155093 +0000 UTC m=+2279.888216937" watchObservedRunningTime="2026-09-09 22:16:26.994288598 +0000 UTC m=+2279.890954608"
Sep 09 22:16:31.360199 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.360168    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"]
Sep 09 22:16:31.364087 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.364068    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.366547 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.366521    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-93d910aa-kserve-self-signed-certs\""
Sep 09 22:16:31.372096 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.372071    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"]
Sep 09 22:16:31.409841 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.409815    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:16:31.469954 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.469926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7781827-3731-4e39-b527-34e15f7824f5-tls-certs\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.470129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.470020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-dshm\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.470129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.470041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-model-cache\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.470129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.470066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-home\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.470129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.470082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-tmp-dir\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.470294 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.470129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqv2s\" (UniqueName: \"kubernetes.io/projected/d7781827-3731-4e39-b527-34e15f7824f5-kube-api-access-gqv2s\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.570889 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.570847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-dshm\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.570889 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.570883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-model-cache\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.571129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.570911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-home\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.571129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.570935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-tmp-dir\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.571129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.570953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqv2s\" (UniqueName: \"kubernetes.io/projected/d7781827-3731-4e39-b527-34e15f7824f5-kube-api-access-gqv2s\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.571129 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.570994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7781827-3731-4e39-b527-34e15f7824f5-tls-certs\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.571367 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.571341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-home\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.571735 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.571709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-model-cache\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.571938 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.571853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-tmp-dir\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.573542 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.573522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-dshm\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.573720 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.573704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7781827-3731-4e39-b527-34e15f7824f5-tls-certs\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.578843 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.578821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqv2s\" (UniqueName: \"kubernetes.io/projected/d7781827-3731-4e39-b527-34e15f7824f5-kube-api-access-gqv2s\") pod \"e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") " pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.674334 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.674250    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:31.948648 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:31.948622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"]
Sep 09 22:16:31.949909 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:16:31.949882    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7781827_3731_4e39_b527_34e15f7824f5.slice/crio-c5156bcad4dfe651cef7299baddc5b53dd1fca233ac5b493a5411ce7c1c9cf82 WatchSource:0}: Error finding container c5156bcad4dfe651cef7299baddc5b53dd1fca233ac5b493a5411ce7c1c9cf82: Status 404 returned error can't find the container with id c5156bcad4dfe651cef7299baddc5b53dd1fca233ac5b493a5411ce7c1c9cf82
Sep 09 22:16:32.003168 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:32.003138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9" event={"ID":"d7781827-3731-4e39-b527-34e15f7824f5","Type":"ContainerStarted","Data":"c5156bcad4dfe651cef7299baddc5b53dd1fca233ac5b493a5411ce7c1c9cf82"}
Sep 09 22:16:33.016842 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:33.016805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9" event={"ID":"d7781827-3731-4e39-b527-34e15f7824f5","Type":"ContainerStarted","Data":"d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32"}
Sep 09 22:16:33.034979 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:33.034936    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9" podStartSLOduration=2.034922534 podStartE2EDuration="2.034922534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:16:33.033872686 +0000 UTC m=+2285.930538693" watchObservedRunningTime="2026-09-09 22:16:33.034922534 +0000 UTC m=+2285.931588542"
Sep 09 22:16:36.286257 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:36.286215    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:36.286680 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:36.286554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:36.304648 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:36.304611    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:37.050466 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:37.050438    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:16:41.674587 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:41.674549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:41.675010 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:41.674598    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:41.687347 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:41.687317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:42.076850 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:42.076817    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:16:55.505394 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:55.505353    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"]
Sep 09 22:16:55.532678 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:55.532648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:16:57.252916 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:57.252882    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"]
Sep 09 22:16:57.283601 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:16:57.283566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:17:03.973712 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:03.973650    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:17:10.525679 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.525628    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p" podUID="aa460355-e76f-4ebb-ad5a-55e3fdc61a69" containerName="main" containerID="cri-o://2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e" gracePeriod=45
Sep 09 22:17:10.783042 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.782976    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:17:10.933625 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.933585    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6zrk\" (UniqueName: \"kubernetes.io/projected/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-kube-api-access-s6zrk\") pod \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") "
Sep 09 22:17:10.933860 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.933634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-model-cache\") pod \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") "
Sep 09 22:17:10.933860 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.933690    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tls-certs\") pod \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") "
Sep 09 22:17:10.933860 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.933706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-home\") pod \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") "
Sep 09 22:17:10.933860 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.933727    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tmp-dir\") pod \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") "
Sep 09 22:17:10.933860 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.933759    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-dshm\") pod \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\" (UID: \"aa460355-e76f-4ebb-ad5a-55e3fdc61a69\") "
Sep 09 22:17:10.934153 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.934037    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa460355-e76f-4ebb-ad5a-55e3fdc61a69" (UID: "aa460355-e76f-4ebb-ad5a-55e3fdc61a69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:17:10.934153 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.934057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-home" (OuterVolumeSpecName: "home") pod "aa460355-e76f-4ebb-ad5a-55e3fdc61a69" (UID: "aa460355-e76f-4ebb-ad5a-55e3fdc61a69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:17:10.934153 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.934069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa460355-e76f-4ebb-ad5a-55e3fdc61a69" (UID: "aa460355-e76f-4ebb-ad5a-55e3fdc61a69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:17:10.936040 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.936015    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-dshm" (OuterVolumeSpecName: "dshm") pod "aa460355-e76f-4ebb-ad5a-55e3fdc61a69" (UID: "aa460355-e76f-4ebb-ad5a-55e3fdc61a69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:17:10.936136 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.936113    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa460355-e76f-4ebb-ad5a-55e3fdc61a69" (UID: "aa460355-e76f-4ebb-ad5a-55e3fdc61a69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:17:10.936189 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:10.936117    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-kube-api-access-s6zrk" (OuterVolumeSpecName: "kube-api-access-s6zrk") pod "aa460355-e76f-4ebb-ad5a-55e3fdc61a69" (UID: "aa460355-e76f-4ebb-ad5a-55e3fdc61a69"). InnerVolumeSpecName "kube-api-access-s6zrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:17:11.035180 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.035089    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:11.035180 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.035119    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:11.035180 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.035133    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:11.035180 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.035143    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:11.035180 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.035153    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:11.035180 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.035163    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6zrk\" (UniqueName: \"kubernetes.io/projected/aa460355-e76f-4ebb-ad5a-55e3fdc61a69-kube-api-access-s6zrk\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:11.229419 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.229385    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa460355-e76f-4ebb-ad5a-55e3fdc61a69" containerID="2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e" exitCode=0
Sep 09 22:17:11.229594 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.229428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"
Sep 09 22:17:11.229594 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.229475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p" event={"ID":"aa460355-e76f-4ebb-ad5a-55e3fdc61a69","Type":"ContainerDied","Data":"2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e"}
Sep 09 22:17:11.229594 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.229512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p" event={"ID":"aa460355-e76f-4ebb-ad5a-55e3fdc61a69","Type":"ContainerDied","Data":"f813d554978aff536939e80ca2b2e5b22f4bed320dd114ef0578ebd0d24dbecd"}
Sep 09 22:17:11.229594 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.229530    2563 scope.go:117] "RemoveContainer" containerID="2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e"
Sep 09 22:17:11.239420 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.239399    2563 scope.go:117] "RemoveContainer" containerID="2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e"
Sep 09 22:17:11.239703 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:17:11.239686    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e\": container with ID starting with 2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e not found: ID does not exist" containerID="2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e"
Sep 09 22:17:11.239749 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.239711    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e"} err="failed to get container status \"2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e\": rpc error: code = NotFound desc = could not find container \"2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e\": container with ID starting with 2efbaa290b22c2910ab146bfee78a27568af73faeddfd1e4e27e598f3b54f00e not found: ID does not exist"
Sep 09 22:17:11.252425 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.252400    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"]
Sep 09 22:17:11.258485 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.258462    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-93d910aa-kserve-5695df8fc9-gxq2p"]
Sep 09 22:17:11.606815 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:11.606764    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa460355-e76f-4ebb-ad5a-55e3fdc61a69" path="/var/lib/kubelet/pods/aa460355-e76f-4ebb-ad5a-55e3fdc61a69/volumes"
Sep 09 22:17:12.273722 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.273683    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9" podUID="d7781827-3731-4e39-b527-34e15f7824f5" containerName="main" containerID="cri-o://d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32" gracePeriod=45
Sep 09 22:17:12.532396 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.532333    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:17:12.646519 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.646493    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-home\") pod \"d7781827-3731-4e39-b527-34e15f7824f5\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") "
Sep 09 22:17:12.646961 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.646590    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-tmp-dir\") pod \"d7781827-3731-4e39-b527-34e15f7824f5\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") "
Sep 09 22:17:12.646961 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.646652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7781827-3731-4e39-b527-34e15f7824f5-tls-certs\") pod \"d7781827-3731-4e39-b527-34e15f7824f5\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") "
Sep 09 22:17:12.646961 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.646713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-model-cache\") pod \"d7781827-3731-4e39-b527-34e15f7824f5\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") "
Sep 09 22:17:12.646961 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.646804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-home" (OuterVolumeSpecName: "home") pod "d7781827-3731-4e39-b527-34e15f7824f5" (UID: "d7781827-3731-4e39-b527-34e15f7824f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:17:12.646961 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.646822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqv2s\" (UniqueName: \"kubernetes.io/projected/d7781827-3731-4e39-b527-34e15f7824f5-kube-api-access-gqv2s\") pod \"d7781827-3731-4e39-b527-34e15f7824f5\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") "
Sep 09 22:17:12.646961 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.646889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-dshm\") pod \"d7781827-3731-4e39-b527-34e15f7824f5\" (UID: \"d7781827-3731-4e39-b527-34e15f7824f5\") "
Sep 09 22:17:12.646961 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.646890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7781827-3731-4e39-b527-34e15f7824f5" (UID: "d7781827-3731-4e39-b527-34e15f7824f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:17:12.646961 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.646961    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7781827-3731-4e39-b527-34e15f7824f5" (UID: "d7781827-3731-4e39-b527-34e15f7824f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:17:12.647337 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.647134    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:12.647337 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.647144    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:12.647337 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.647152    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:12.649013 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.648982    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7781827-3731-4e39-b527-34e15f7824f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7781827-3731-4e39-b527-34e15f7824f5" (UID: "d7781827-3731-4e39-b527-34e15f7824f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:17:12.649124 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.649038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-dshm" (OuterVolumeSpecName: "dshm") pod "d7781827-3731-4e39-b527-34e15f7824f5" (UID: "d7781827-3731-4e39-b527-34e15f7824f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:17:12.649124 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.649058    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7781827-3731-4e39-b527-34e15f7824f5-kube-api-access-gqv2s" (OuterVolumeSpecName: "kube-api-access-gqv2s") pod "d7781827-3731-4e39-b527-34e15f7824f5" (UID: "d7781827-3731-4e39-b527-34e15f7824f5"). InnerVolumeSpecName "kube-api-access-gqv2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:17:12.747823 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.747760    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7781827-3731-4e39-b527-34e15f7824f5-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:12.747823 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.747815    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqv2s\" (UniqueName: \"kubernetes.io/projected/d7781827-3731-4e39-b527-34e15f7824f5-kube-api-access-gqv2s\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:12.747823 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:12.747824    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7781827-3731-4e39-b527-34e15f7824f5-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:13.244822 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:13.244755    2563 generic.go:356] "Generic (PLEG): container finished" podID="d7781827-3731-4e39-b527-34e15f7824f5" containerID="d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32" exitCode=0
Sep 09 22:17:13.244822 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:13.244820    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"
Sep 09 22:17:13.245149 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:13.244863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9" event={"ID":"d7781827-3731-4e39-b527-34e15f7824f5","Type":"ContainerDied","Data":"d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32"}
Sep 09 22:17:13.245149 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:13.244903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9" event={"ID":"d7781827-3731-4e39-b527-34e15f7824f5","Type":"ContainerDied","Data":"c5156bcad4dfe651cef7299baddc5b53dd1fca233ac5b493a5411ce7c1c9cf82"}
Sep 09 22:17:13.245149 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:13.244920    2563 scope.go:117] "RemoveContainer" containerID="d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32"
Sep 09 22:17:13.257311 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:13.257287    2563 scope.go:117] "RemoveContainer" containerID="d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32"
Sep 09 22:17:13.257603 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:17:13.257582    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32\": container with ID starting with d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32 not found: ID does not exist" containerID="d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32"
Sep 09 22:17:13.257691 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:13.257622    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32"} err="failed to get container status \"d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32\": rpc error: code = NotFound desc = could not find container \"d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32\": container with ID starting with d807463bdf0b844fd07e5b11338991c196fb5154fa8d18acbfaba487c34e9d32 not found: ID does not exist"
Sep 09 22:17:13.266777 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:13.266740    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"]
Sep 09 22:17:13.269885 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:13.269860    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-93d910aa-kserve-5b487dbbd7-k56d9"]
Sep 09 22:17:13.606705 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:13.606624    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7781827-3731-4e39-b527-34e15f7824f5" path="/var/lib/kubelet/pods/d7781827-3731-4e39-b527-34e15f7824f5/volumes"
Sep 09 22:17:30.543741 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.543711    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"]
Sep 09 22:17:30.544168 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.544123    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa460355-e76f-4ebb-ad5a-55e3fdc61a69" containerName="main"
Sep 09 22:17:30.544168 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.544134    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa460355-e76f-4ebb-ad5a-55e3fdc61a69" containerName="main"
Sep 09 22:17:30.544168 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.544151    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7781827-3731-4e39-b527-34e15f7824f5" containerName="main"
Sep 09 22:17:30.544168 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.544156    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7781827-3731-4e39-b527-34e15f7824f5" containerName="main"
Sep 09 22:17:30.544304 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.544221    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7781827-3731-4e39-b527-34e15f7824f5" containerName="main"
Sep 09 22:17:30.544304 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.544229    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa460355-e76f-4ebb-ad5a-55e3fdc61a69" containerName="main"
Sep 09 22:17:30.549092 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.549074    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.555438 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.555408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"]
Sep 09 22:17:30.616404 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.616368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk26w\" (UniqueName: \"kubernetes.io/projected/d7e778cf-c3b4-4d96-935d-e1831181b474-kube-api-access-jk26w\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.616582 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.616434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e778cf-c3b4-4d96-935d-e1831181b474-maas-api-tls\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.616582 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.616451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e778cf-c3b4-4d96-935d-e1831181b474-metrics-certs\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.616582 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.616481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e778cf-c3b4-4d96-935d-e1831181b474-openshift-service-ca\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.717611 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.717575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e778cf-c3b4-4d96-935d-e1831181b474-maas-api-tls\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.717611 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.717619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e778cf-c3b4-4d96-935d-e1831181b474-metrics-certs\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.717889 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.717672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e778cf-c3b4-4d96-935d-e1831181b474-openshift-service-ca\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.717889 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.717718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk26w\" (UniqueName: \"kubernetes.io/projected/d7e778cf-c3b4-4d96-935d-e1831181b474-kube-api-access-jk26w\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.718357 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.718332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e778cf-c3b4-4d96-935d-e1831181b474-openshift-service-ca\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.720332 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.720310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e778cf-c3b4-4d96-935d-e1831181b474-metrics-certs\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.720399 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.720310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e778cf-c3b4-4d96-935d-e1831181b474-maas-api-tls\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.725560 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.725533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk26w\" (UniqueName: \"kubernetes.io/projected/d7e778cf-c3b4-4d96-935d-e1831181b474-kube-api-access-jk26w\") pod \"maas-api-77fc54c768-8sp45\" (UID: \"d7e778cf-c3b4-4d96-935d-e1831181b474\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:30.862663 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:30.862556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:31.135221 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:31.135190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"]
Sep 09 22:17:31.136228 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:17:31.136202    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e778cf_c3b4_4d96_935d_e1831181b474.slice/crio-4b9e0bc8361d98b2de6854781ad08091c5f3da1494b14171b45324fbd26a58a6 WatchSource:0}: Error finding container 4b9e0bc8361d98b2de6854781ad08091c5f3da1494b14171b45324fbd26a58a6: Status 404 returned error can't find the container with id 4b9e0bc8361d98b2de6854781ad08091c5f3da1494b14171b45324fbd26a58a6
Sep 09 22:17:31.345157 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:31.345126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45" event={"ID":"d7e778cf-c3b4-4d96-935d-e1831181b474","Type":"ContainerStarted","Data":"4b9e0bc8361d98b2de6854781ad08091c5f3da1494b14171b45324fbd26a58a6"}
Sep 09 22:17:32.352929 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:32.352895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45" event={"ID":"d7e778cf-c3b4-4d96-935d-e1831181b474","Type":"ContainerStarted","Data":"46b5bb80348fee08e68812168269e29ddde5a239cfbe038446e9782e07f267bb"}
Sep 09 22:17:32.353339 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:32.352995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:32.371500 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:32.371448    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45" podStartSLOduration=1.645000761 podStartE2EDuration="2.371432867s" totalImagesPullingTime="726.432106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:17:32.370108601 +0000 UTC m=+2345.266774610" watchObservedRunningTime="2026-09-09 22:17:32.371432867 +0000 UTC m=+2345.268098906"
Sep 09 22:17:38.366095 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:38.366066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77fc54c768-8sp45"
Sep 09 22:17:38.415956 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:38.415914    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"]
Sep 09 22:17:38.416197 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:38.416154    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r" podUID="d06ed691-a797-4a06-b9be-e08cfda44752" containerName="maas-api" containerID="cri-o://5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8" gracePeriod=30
Sep 09 22:17:39.662795 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.662750    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 22:17:39.804132 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.804098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06ed691-a797-4a06-b9be-e08cfda44752-openshift-service-ca\") pod \"d06ed691-a797-4a06-b9be-e08cfda44752\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") "
Sep 09 22:17:39.804132 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.804134    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpbvk\" (UniqueName: \"kubernetes.io/projected/d06ed691-a797-4a06-b9be-e08cfda44752-kube-api-access-tpbvk\") pod \"d06ed691-a797-4a06-b9be-e08cfda44752\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") "
Sep 09 22:17:39.804399 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.804154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-maas-api-tls\") pod \"d06ed691-a797-4a06-b9be-e08cfda44752\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") "
Sep 09 22:17:39.804399 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.804212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-metrics-certs\") pod \"d06ed691-a797-4a06-b9be-e08cfda44752\" (UID: \"d06ed691-a797-4a06-b9be-e08cfda44752\") "
Sep 09 22:17:39.804564 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.804538    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06ed691-a797-4a06-b9be-e08cfda44752-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d06ed691-a797-4a06-b9be-e08cfda44752" (UID: "d06ed691-a797-4a06-b9be-e08cfda44752"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:17:39.806560 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.806528    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d06ed691-a797-4a06-b9be-e08cfda44752" (UID: "d06ed691-a797-4a06-b9be-e08cfda44752"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:17:39.806682 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.806564    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06ed691-a797-4a06-b9be-e08cfda44752-kube-api-access-tpbvk" (OuterVolumeSpecName: "kube-api-access-tpbvk") pod "d06ed691-a797-4a06-b9be-e08cfda44752" (UID: "d06ed691-a797-4a06-b9be-e08cfda44752"). InnerVolumeSpecName "kube-api-access-tpbvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:17:39.806682 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.806574    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d06ed691-a797-4a06-b9be-e08cfda44752" (UID: "d06ed691-a797-4a06-b9be-e08cfda44752"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:17:39.905223 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.905189    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06ed691-a797-4a06-b9be-e08cfda44752-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:39.905223 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.905219    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpbvk\" (UniqueName: \"kubernetes.io/projected/d06ed691-a797-4a06-b9be-e08cfda44752-kube-api-access-tpbvk\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:39.905223 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.905229    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:39.905433 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:39.905237    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d06ed691-a797-4a06-b9be-e08cfda44752-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 22:17:40.400199 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:40.400164    2563 generic.go:356] "Generic (PLEG): container finished" podID="d06ed691-a797-4a06-b9be-e08cfda44752" containerID="5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8" exitCode=0
Sep 09 22:17:40.400388 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:40.400211    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"
Sep 09 22:17:40.400388 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:40.400256    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r" event={"ID":"d06ed691-a797-4a06-b9be-e08cfda44752","Type":"ContainerDied","Data":"5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8"}
Sep 09 22:17:40.400388 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:40.400316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r" event={"ID":"d06ed691-a797-4a06-b9be-e08cfda44752","Type":"ContainerDied","Data":"ebcf50471a3d1f11f4280302fba4bbbc1924692e09cf1366d15561e099aa1cfe"}
Sep 09 22:17:40.400388 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:40.400346    2563 scope.go:117] "RemoveContainer" containerID="5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8"
Sep 09 22:17:40.410530 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:40.410510    2563 scope.go:117] "RemoveContainer" containerID="5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8"
Sep 09 22:17:40.410829 ip-10-0-129-45 kubenswrapper[2563]: E0909 22:17:40.410810    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8\": container with ID starting with 5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8 not found: ID does not exist" containerID="5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8"
Sep 09 22:17:40.410881 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:40.410840    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8"} err="failed to get container status \"5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8\": rpc error: code = NotFound desc = could not find container \"5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8\": container with ID starting with 5e9cf0974ddab132acbbeae806b10ab73629a096e3a35674283fc2e383421ee8 not found: ID does not exist"
Sep 09 22:17:40.422697 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:40.422678    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"]
Sep 09 22:17:40.426502 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:40.426478    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-77bd4d9968-j8f8r"]
Sep 09 22:17:41.605640 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:41.605592    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d06ed691-a797-4a06-b9be-e08cfda44752" path="/var/lib/kubelet/pods/d06ed691-a797-4a06-b9be-e08cfda44752/volumes"
Sep 09 22:17:50.318391 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:50.318352    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b7f69cd5-928jh_dde1498b-bf96-43df-b65a-6a6cd5080c00/authorino/0.log"
Sep 09 22:17:55.721929 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:55.721893    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c999c7bf-9jr7t_6c9d113a-6c44-4e06-8d67-210126253147/manager/0.log"
Sep 09 22:17:55.825331 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:55.825302    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zmqhp_979bdb69-b60e-49a6-92f0-35cc3746740c/server/0.log"
Sep 09 22:17:55.942961 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:55.942894    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-l5s9q_6a6767bd-31c2-48f0-bacd-b29eeb1a47d9/manager/0.log"
Sep 09 22:17:56.315030 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:56.314982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56bc947947-stwkq_99905997-d07a-4e23-9a27-67c395cf1347/manager/0.log"
Sep 09 22:17:56.322521 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:56.322497    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56bc947947-stwkq_99905997-d07a-4e23-9a27-67c395cf1347/copy-manifests/0.log"
Sep 09 22:17:56.670235 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:56.670119    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-77fc54c768-8sp45_d7e778cf-c3b4-4d96-935d-e1831181b474/maas-api/0.log"
Sep 09 22:17:57.944573 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:57.944540    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b7f69cd5-928jh_dde1498b-bf96-43df-b65a-6a6cd5080c00/authorino/0.log"
Sep 09 22:17:59.184078 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:59.184040    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b49d8b8f6-m54ns_626afafe-e25d-4f41-b58f-86cf0cf0df6e/kube-auth-proxy/0.log"
Sep 09 22:17:59.661649 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:17:59.661608    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9pcwv_d5b4be3d-5973-4de5-ad82-0582fb44a438/payload-processing/0.log"
Sep 09 22:18:00.541199 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:00.541151    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpg6df_3ea97abd-538f-4db7-9fe7-c52cce8a345e/main/0.log"
Sep 09 22:18:05.785003 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.784965    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pjhm4/must-gather-qf7w8"]
Sep 09 22:18:05.785400 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.785356    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06ed691-a797-4a06-b9be-e08cfda44752" containerName="maas-api"
Sep 09 22:18:05.785400 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.785368    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06ed691-a797-4a06-b9be-e08cfda44752" containerName="maas-api"
Sep 09 22:18:05.785470 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.785430    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06ed691-a797-4a06-b9be-e08cfda44752" containerName="maas-api"
Sep 09 22:18:05.788431 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.788410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjhm4/must-gather-qf7w8"
Sep 09 22:18:05.790995 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.790969    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pjhm4\"/\"default-dockercfg-627h7\""
Sep 09 22:18:05.791089 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.791006    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjhm4\"/\"kube-root-ca.crt\""
Sep 09 22:18:05.791089 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.790975    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjhm4\"/\"openshift-service-ca.crt\""
Sep 09 22:18:05.796668 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.796547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:18:05.797463 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.797438    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjhm4/must-gather-qf7w8"]
Sep 09 22:18:05.840119 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.840093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzn4d\" (UniqueName: \"kubernetes.io/projected/75a4ca19-6007-4c1f-8241-869a08ab7d9b-kube-api-access-zzn4d\") pod \"must-gather-qf7w8\" (UID: \"75a4ca19-6007-4c1f-8241-869a08ab7d9b\") " pod="openshift-must-gather-pjhm4/must-gather-qf7w8"
Sep 09 22:18:05.840265 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.840130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75a4ca19-6007-4c1f-8241-869a08ab7d9b-must-gather-output\") pod \"must-gather-qf7w8\" (UID: \"75a4ca19-6007-4c1f-8241-869a08ab7d9b\") " pod="openshift-must-gather-pjhm4/must-gather-qf7w8"
Sep 09 22:18:05.896932 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.896891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:18:05.941322 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.941288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzn4d\" (UniqueName: \"kubernetes.io/projected/75a4ca19-6007-4c1f-8241-869a08ab7d9b-kube-api-access-zzn4d\") pod \"must-gather-qf7w8\" (UID: \"75a4ca19-6007-4c1f-8241-869a08ab7d9b\") " pod="openshift-must-gather-pjhm4/must-gather-qf7w8"
Sep 09 22:18:05.941469 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.941334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75a4ca19-6007-4c1f-8241-869a08ab7d9b-must-gather-output\") pod \"must-gather-qf7w8\" (UID: \"75a4ca19-6007-4c1f-8241-869a08ab7d9b\") " pod="openshift-must-gather-pjhm4/must-gather-qf7w8"
Sep 09 22:18:05.941648 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.941633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75a4ca19-6007-4c1f-8241-869a08ab7d9b-must-gather-output\") pod \"must-gather-qf7w8\" (UID: \"75a4ca19-6007-4c1f-8241-869a08ab7d9b\") " pod="openshift-must-gather-pjhm4/must-gather-qf7w8"
Sep 09 22:18:05.949445 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:05.949416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzn4d\" (UniqueName: \"kubernetes.io/projected/75a4ca19-6007-4c1f-8241-869a08ab7d9b-kube-api-access-zzn4d\") pod \"must-gather-qf7w8\" (UID: \"75a4ca19-6007-4c1f-8241-869a08ab7d9b\") " pod="openshift-must-gather-pjhm4/must-gather-qf7w8"
Sep 09 22:18:06.098894 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:06.098794    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjhm4/must-gather-qf7w8"
Sep 09 22:18:06.375343 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:06.375318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjhm4/must-gather-qf7w8"]
Sep 09 22:18:06.377726 ip-10-0-129-45 kubenswrapper[2563]: W0909 22:18:06.377695    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75a4ca19_6007_4c1f_8241_869a08ab7d9b.slice/crio-bc9da0d6c285fb8392526f7d6010048832380b35c0392892ee6ef1599acf0404 WatchSource:0}: Error finding container bc9da0d6c285fb8392526f7d6010048832380b35c0392892ee6ef1599acf0404: Status 404 returned error can't find the container with id bc9da0d6c285fb8392526f7d6010048832380b35c0392892ee6ef1599acf0404
Sep 09 22:18:06.379951 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:06.379934    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 22:18:06.559824 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:06.559790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjhm4/must-gather-qf7w8" event={"ID":"75a4ca19-6007-4c1f-8241-869a08ab7d9b","Type":"ContainerStarted","Data":"bc9da0d6c285fb8392526f7d6010048832380b35c0392892ee6ef1599acf0404"}
Sep 09 22:18:07.567110 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:07.567068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjhm4/must-gather-qf7w8" event={"ID":"75a4ca19-6007-4c1f-8241-869a08ab7d9b","Type":"ContainerStarted","Data":"b605a943e3ad6048a67502859246af4fd3db0bcee619fb5399e89c514a159f4f"}
Sep 09 22:18:07.567504 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:07.567128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjhm4/must-gather-qf7w8" event={"ID":"75a4ca19-6007-4c1f-8241-869a08ab7d9b","Type":"ContainerStarted","Data":"0ce1d64d36eab75d56b6c34d84164b92e03ced9b783b1e60c25127875f5d0319"}
Sep 09 22:18:07.587475 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:07.587406    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pjhm4/must-gather-qf7w8" podStartSLOduration=1.7230596459999998 podStartE2EDuration="2.587383021s" totalImagesPullingTime="864.323375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:18:07.584405595 +0000 UTC m=+2380.481071606" watchObservedRunningTime="2026-09-09 22:18:07.587383021 +0000 UTC m=+2380.484049032"
Sep 09 22:18:07.730285 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:07.730234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:18:07.786608 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:07.786568    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log" limit=1024
Sep 09 22:18:08.874404 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:08.874355    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nd9gr_6271f963-afc0-421e-8d4f-cbe5eedbc4d8/global-pull-secret-syncer/0.log"
Sep 09 22:18:08.966643 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:08.966617    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5npzg_c9b64455-a274-4e65-91b6-849f97952b2a/konnectivity-agent/0.log"
Sep 09 22:18:09.047718 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:09.047689    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-45.ec2.internal_1034d32653d0ee36541005678f807d7f/haproxy/0.log"
Sep 09 22:18:13.792525 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:13.792493    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b7f69cd5-928jh_dde1498b-bf96-43df-b65a-6a6cd5080c00/authorino/0.log"
Sep 09 22:18:15.644388 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:15.644348    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-8pcxl_6ab3349e-53f6-462b-a888-bff0d463fd2f/cluster-monitoring-operator/0.log"
Sep 09 22:18:15.771543 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:15.771502    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-mqp2s_60c67e91-3ce5-4a19-bb28-26afe4d49baa/monitoring-plugin/0.log"
Sep 09 22:18:15.803399 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:15.803362    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ghzwq_f7714c74-de4f-4360-a16a-5a68938045c6/node-exporter/0.log"
Sep 09 22:18:15.824852 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:15.824816    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ghzwq_f7714c74-de4f-4360-a16a-5a68938045c6/kube-rbac-proxy/0.log"
Sep 09 22:18:15.845450 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:15.845417    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ghzwq_f7714c74-de4f-4360-a16a-5a68938045c6/init-textfile/0.log"
Sep 09 22:18:16.354848 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.354739    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-tcx77_85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1/prometheus-operator/0.log"
Sep 09 22:18:16.373721 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.373688    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-tcx77_85f25c3d-9d3e-4c3f-bf65-5f6b70ab74c1/kube-rbac-proxy/0.log"
Sep 09 22:18:16.398016 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.397986    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-f2p52_e9842591-f7c3-4a3c-be71-d425623e63a2/prometheus-operator-admission-webhook/0.log"
Sep 09 22:18:16.430277 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.430247    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6989ff7f7d-dhzpt_d8fa0a29-c6f4-48c3-8096-a67459be5ce4/telemeter-client/0.log"
Sep 09 22:18:16.450908 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.450871    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6989ff7f7d-dhzpt_d8fa0a29-c6f4-48c3-8096-a67459be5ce4/reload/0.log"
Sep 09 22:18:16.476990 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.476961    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6989ff7f7d-dhzpt_d8fa0a29-c6f4-48c3-8096-a67459be5ce4/kube-rbac-proxy/0.log"
Sep 09 22:18:16.508708 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.508676    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7869998cc7-q2cj4_e7e916f3-911a-4310-b1a0-6bae8cfeee77/thanos-query/0.log"
Sep 09 22:18:16.532556 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.532518    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7869998cc7-q2cj4_e7e916f3-911a-4310-b1a0-6bae8cfeee77/kube-rbac-proxy-web/0.log"
Sep 09 22:18:16.557639 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.557596    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7869998cc7-q2cj4_e7e916f3-911a-4310-b1a0-6bae8cfeee77/kube-rbac-proxy/0.log"
Sep 09 22:18:16.581057 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.581001    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7869998cc7-q2cj4_e7e916f3-911a-4310-b1a0-6bae8cfeee77/prom-label-proxy/0.log"
Sep 09 22:18:16.606425 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.606333    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7869998cc7-q2cj4_e7e916f3-911a-4310-b1a0-6bae8cfeee77/kube-rbac-proxy-rules/0.log"
Sep 09 22:18:16.633138 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:16.633101    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7869998cc7-q2cj4_e7e916f3-911a-4310-b1a0-6bae8cfeee77/kube-rbac-proxy-metrics/0.log"
Sep 09 22:18:18.100623 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:18.100592    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-vs5kp_f742f48a-2f95-4121-adaf-4a1f023498df/networking-console-plugin/0.log"
Sep 09 22:18:18.697244 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:18.697213    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 22:18:18.705527 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:18.705499    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/2.log"
Sep 09 22:18:19.218040 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.218002    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d797bc5b8-r8dpf_79964bc9-ba89-4adf-bf5e-6ee2ac038b15/console/0.log"
Sep 09 22:18:19.251252 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.251215    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-cwvcs_332f31a4-50a6-4169-9b12-c76283c2f7d4/download-server/0.log"
Sep 09 22:18:19.629381 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.629344    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"]
Sep 09 22:18:19.639268 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.637584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.646078 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.643945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"]
Sep 09 22:18:19.689641 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.689597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-proc\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.689848 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.689652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-lib-modules\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.689848 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.689685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-sys\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.689848 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.689804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-podres\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.689963 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.689882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzcrf\" (UniqueName: \"kubernetes.io/projected/846feb7a-2c34-45f1-b04c-7bb190245c49-kube-api-access-rzcrf\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.790657 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.790618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzcrf\" (UniqueName: \"kubernetes.io/projected/846feb7a-2c34-45f1-b04c-7bb190245c49-kube-api-access-rzcrf\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.790902 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.790710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-proc\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.790902 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.790743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-lib-modules\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.790902 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.790817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-sys\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.790902 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.790881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-podres\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.790902 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.790887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-proc\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.791198 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.791009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-sys\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.791198 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.791105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-podres\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.791198 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.791112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/846feb7a-2c34-45f1-b04c-7bb190245c49-lib-modules\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.805434 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.805404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzcrf\" (UniqueName: \"kubernetes.io/projected/846feb7a-2c34-45f1-b04c-7bb190245c49-kube-api-access-rzcrf\") pod \"perf-node-gather-daemonset-2ks7r\" (UID: \"846feb7a-2c34-45f1-b04c-7bb190245c49\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:19.930075 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.929995    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-gmnwj_b903bce2-59e5-4b6f-9cb5-ebea1800e44f/volume-data-source-validator/0.log"
Sep 09 22:18:19.953075 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:19.953038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:20.256211 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:20.256169    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"]
Sep 09 22:18:20.695854 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:20.695747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r" event={"ID":"846feb7a-2c34-45f1-b04c-7bb190245c49","Type":"ContainerStarted","Data":"33623baa9f404f42a25c29dadccfd5eaa299814676f36c77ad99cd256724b972"}
Sep 09 22:18:20.696123 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:20.695863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r" event={"ID":"846feb7a-2c34-45f1-b04c-7bb190245c49","Type":"ContainerStarted","Data":"39eaeeb4c5d9e372c26d409e99b5f64bc146ca3c9d9bbaefc93222aecdf671ab"}
Sep 09 22:18:20.696716 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:20.696690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:20.717678 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:20.717591    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r" podStartSLOduration=1.717570926 podStartE2EDuration="1.717570926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:18:20.714859343 +0000 UTC m=+2393.611525376" watchObservedRunningTime="2026-09-09 22:18:20.717570926 +0000 UTC m=+2393.614236932"
Sep 09 22:18:21.180246 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:21.180217    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zh6lr_888f9b50-262e-4525-a9f2-3857190be163/dns/0.log"
Sep 09 22:18:21.198970 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:21.198938    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zh6lr_888f9b50-262e-4525-a9f2-3857190be163/kube-rbac-proxy/0.log"
Sep 09 22:18:21.235245 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:21.235213    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-68g9z_d7198ae4-9d2d-41cb-892c-90e8f729c20e/dns-node-resolver/0.log"
Sep 09 22:18:21.840236 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:21.840208    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tlfbl_715cecb6-2493-4cd4-84b8-6eb84e62fb1b/node-ca/0.log"
Sep 09 22:18:22.984503 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:22.984469    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b49d8b8f6-m54ns_626afafe-e25d-4f41-b58f-86cf0cf0df6e/kube-auth-proxy/0.log"
Sep 09 22:18:23.126470 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:23.126432    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9pcwv_d5b4be3d-5973-4de5-ad82-0582fb44a438/payload-processing/0.log"
Sep 09 22:18:24.042123 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:24.042084    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n6cbw_0b8f9430-2b08-481b-9d13-4290193c0092/serve-healthcheck-canary/0.log"
Sep 09 22:18:24.525685 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:24.525658    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48qfx_9de15aa3-26a6-4d46-89fe-c08d0b6a853e/kube-rbac-proxy/0.log"
Sep 09 22:18:24.548554 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:24.548519    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48qfx_9de15aa3-26a6-4d46-89fe-c08d0b6a853e/exporter/0.log"
Sep 09 22:18:24.568205 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:24.568178    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48qfx_9de15aa3-26a6-4d46-89fe-c08d0b6a853e/extractor/0.log"
Sep 09 22:18:27.093578 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:27.093542    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c999c7bf-9jr7t_6c9d113a-6c44-4e06-8d67-210126253147/manager/0.log"
Sep 09 22:18:27.121083 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:27.121047    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zmqhp_979bdb69-b60e-49a6-92f0-35cc3746740c/server/0.log"
Sep 09 22:18:27.264028 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:27.263987    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-l5s9q_6a6767bd-31c2-48f0-bacd-b29eeb1a47d9/manager/0.log"
Sep 09 22:18:27.469976 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:27.469876    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56bc947947-stwkq_99905997-d07a-4e23-9a27-67c395cf1347/manager/0.log"
Sep 09 22:18:27.484989 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:27.484964    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56bc947947-stwkq_99905997-d07a-4e23-9a27-67c395cf1347/copy-manifests/0.log"
Sep 09 22:18:27.722530 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:27.722443    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-2ks7r"
Sep 09 22:18:27.796677 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:27.794817    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 22:18:27.811566 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:27.811528    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z2v2b_448a39a8-aab5-4696-b385-2169e6ee4a6e/console-operator/1.log"
Sep 09 22:18:28.653905 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:28.653864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-798b8d4d4-wq4x7_f4e5b599-6060-4e81-88bd-574c218c7bb1/manager/0.log"
Sep 09 22:18:29.783539 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:29.783506    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b49d8b8f6-m54ns_626afafe-e25d-4f41-b58f-86cf0cf0df6e/kube-auth-proxy/0.log"
Sep 09 22:18:29.890004 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:29.889973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9pcwv_d5b4be3d-5973-4de5-ad82-0582fb44a438/payload-processing/0.log"
Sep 09 22:18:31.722532 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:31.722498    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b49d8b8f6-m54ns_626afafe-e25d-4f41-b58f-86cf0cf0df6e/kube-auth-proxy/0.log"
Sep 09 22:18:31.829783 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:31.829736    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9pcwv_d5b4be3d-5973-4de5-ad82-0582fb44a438/payload-processing/0.log"
Sep 09 22:18:35.681053 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:35.681020    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnfn6_da6de3d9-7174-44f8-9773-fc7f5b353bd8/kube-multus-additional-cni-plugins/0.log"
Sep 09 22:18:35.700602 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:35.700571    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnfn6_da6de3d9-7174-44f8-9773-fc7f5b353bd8/egress-router-binary-copy/0.log"
Sep 09 22:18:35.719520 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:35.719494    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnfn6_da6de3d9-7174-44f8-9773-fc7f5b353bd8/cni-plugins/0.log"
Sep 09 22:18:35.739509 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:35.739484    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnfn6_da6de3d9-7174-44f8-9773-fc7f5b353bd8/bond-cni-plugin/0.log"
Sep 09 22:18:35.757995 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:35.757973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnfn6_da6de3d9-7174-44f8-9773-fc7f5b353bd8/routeoverride-cni/0.log"
Sep 09 22:18:35.775590 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:35.775568    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnfn6_da6de3d9-7174-44f8-9773-fc7f5b353bd8/whereabouts-cni-bincopy/0.log"
Sep 09 22:18:35.794804 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:35.794737    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnfn6_da6de3d9-7174-44f8-9773-fc7f5b353bd8/whereabouts-cni/0.log"
Sep 09 22:18:35.861419 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:35.861389    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-krpdt_5aca247b-df86-41d9-9b9d-c9e302f8df71/kube-multus/0.log"
Sep 09 22:18:35.926824 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:35.926796    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vsdv8_f9a1488a-6bce-4e7b-846d-37825f27b619/network-metrics-daemon/0.log"
Sep 09 22:18:35.941321 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:35.941252    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vsdv8_f9a1488a-6bce-4e7b-846d-37825f27b619/kube-rbac-proxy/0.log"
Sep 09 22:18:37.271303 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:37.271275    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zchqg_773f5b77-c88f-4e79-8104-fd4f9bc8c196/ovn-controller/0.log"
Sep 09 22:18:37.286528 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:37.286508    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zchqg_773f5b77-c88f-4e79-8104-fd4f9bc8c196/ovn-acl-logging/0.log"
Sep 09 22:18:37.314604 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:37.314576    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zchqg_773f5b77-c88f-4e79-8104-fd4f9bc8c196/kube-rbac-proxy-node/0.log"
Sep 09 22:18:37.339864 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:37.339836    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zchqg_773f5b77-c88f-4e79-8104-fd4f9bc8c196/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 22:18:37.357495 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:37.357460    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zchqg_773f5b77-c88f-4e79-8104-fd4f9bc8c196/northd/0.log"
Sep 09 22:18:37.376116 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:37.376096    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zchqg_773f5b77-c88f-4e79-8104-fd4f9bc8c196/nbdb/0.log"
Sep 09 22:18:37.395821 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:37.395797    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zchqg_773f5b77-c88f-4e79-8104-fd4f9bc8c196/sbdb/0.log"
Sep 09 22:18:37.490386 ip-10-0-129-45 kubenswrapper[2563]: I0909 22:18:37.490355    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zchqg_773f5b77-c88f-4e79-8104-fd4f9bc8c196/ovnkube-controller/0.log"