Sep 11 18:13:10.007830 ip-10-0-129-7 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 18:13:10.007843 ip-10-0-129-7 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 18:13:10.007853 ip-10-0-129-7 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 18:13:10.008085 ip-10-0-129-7 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 18:13:21.226723 ip-10-0-129-7 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 18:13:21.226737 ip-10-0-129-7 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 01fd36a69ef940bbbdddd7a0941afd46 --
Sep 11 18:15:18.790560 ip-10-0-129-7 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 18:15:19.187212 ip-10-0-129-7 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:19.187212 ip-10-0-129-7 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 18:15:19.187212 ip-10-0-129-7 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:19.187212 ip-10-0-129-7 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 18:15:19.187212 ip-10-0-129-7 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:19.188406 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.188307    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 18:15:19.192089 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192055    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:19.192089 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192088    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:19.192089 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192092    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192096    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192099    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192103    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192106    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192108    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192111    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192114    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192116    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192119    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192122    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192125    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192127    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192130    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192133    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192136    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192140    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192143    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192146    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192149    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:19.192192 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192152    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192155    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192157    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192160    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192162    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192165    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192167    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192170    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192172    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192175    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192177    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192180    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192182    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192185    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192188    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192191    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192208    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192213    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192216    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192219    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:19.192694 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192222    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192225    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192228    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192231    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192234    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192236    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192239    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192241    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192244    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192247    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192250    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192252    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192255    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192257    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192260    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192263    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192267    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192269    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192273    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:19.193248 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192275    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192279    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192283    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192286    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192289    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192291    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192294    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192298    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192301    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192305    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192308    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192311    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192317    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192321    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192324    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192327    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192330    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192333    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192336    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:19.193721 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192339    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192342    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192344    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192347    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192350    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192352    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192355    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192358    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192360    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192363    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192366    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192368    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192371    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192374    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192377    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192379    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192382    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192384    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.192387    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193138    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:19.194214 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193147    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193151    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193154    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193158    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193162    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193164    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193167    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193170    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193173    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193176    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193179    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193181    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193184    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193187    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193189    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193192    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193212    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193215    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193218    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:19.194716 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193221    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193224    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193227    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193230    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193233    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193236    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193238    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193241    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193244    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193247    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193250    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193252    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193255    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193261    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193265    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193268    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193271    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193273    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193278    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193282    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:19.195187 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193285    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193288    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193292    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193294    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193298    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193301    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193304    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193306    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193310    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193313    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193316    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193318    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193321    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193323    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193327    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193331    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193334    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193337    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193340    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:19.195737 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193342    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193346    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193350    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193353    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193355    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193358    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193361    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193364    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193367    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193370    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193372    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193375    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193378    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193381    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193384    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193386    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193389    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193392    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193394    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193397    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:19.196231 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193399    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193402    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193404    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193407    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193409    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193412    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193415    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193417    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193420    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193422    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193425    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193427    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193430    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193432    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193435    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193438    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193440    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193443    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193446    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193449    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:19.196740 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193854    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193859    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193864    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193868    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193874    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193877    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193881    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193884    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193887    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193890    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193893    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193896    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193899    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193902    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193905    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193908    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193911    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193913    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193916    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193919    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:19.197254 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193921    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193924    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193927    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193929    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193931    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193934    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193937    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193939    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193942    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193945    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193948    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193951    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193954    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193957    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193960    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193962    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193965    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193968    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193971    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:19.197752 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193974    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193976    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193979    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193982    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193984    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193987    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193990    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193992    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193995    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.193997    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194000    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194003    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194006    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194008    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194011    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194013    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194016    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194018    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194021    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194023    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:19.198225 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194026    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194028    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194031    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194034    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194037    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194040    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194042    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194045    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194047    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194050    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194052    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194055    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194059    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194063    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194065    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194068    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194070    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194073    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194075    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:19.198747 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194078    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194080    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194083    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194088    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194093    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194095    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194098    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194101    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194103    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194106    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194108    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194111    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194114    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194118    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194121    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194124    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194127    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194130    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194133    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:19.199245 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194135    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194138    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194233    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194271    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194277    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194282    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194288    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194291    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194296    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194300    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194303    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194306    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194310    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194313    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194316    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194320    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194325    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194329    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194332    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194335    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194340    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194342    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194346    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194349    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 18:15:19.199720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194352    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194356    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194359    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194362    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194365    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194369    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194371    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194374    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194377    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194380    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194384    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194388    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194391    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194393    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194396    2580 flags.go:64] FLAG: --enable-server="true"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194400    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194405    2580 flags.go:64] FLAG: --event-burst="100"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194408    2580 flags.go:64] FLAG: --event-qps="50"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194411    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194413    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194417    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194421    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194424    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194427    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194431    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 11 18:15:19.200339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194436    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194439    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194442    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194445    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194448    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194451    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194454    2580 flags.go:64] FLAG: --feature-gates=""
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194458    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194461    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194464    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194468    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194471    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194474    2580 flags.go:64] FLAG: --help="false"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194477    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194480    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194484    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194487    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194490    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194494    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194497    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194500    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194502    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194505    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 18:15:19.201043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194509    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194513    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194516    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194519    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194521    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194525    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194527    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194530    2580 flags.go:64] FLAG: --lock-file=""
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194535    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194539    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194542    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194548    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194551    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194554    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194557    2580 flags.go:64] FLAG: --logging-format="text"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194560    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194563    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194566    2580 flags.go:64] FLAG: --manifest-url=""
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194569    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194574    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194577    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194581    2580 flags.go:64] FLAG: --max-pods="110"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194584    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194587    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194590    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 18:15:19.201652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194593    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194596    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194599    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194602    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194609    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194612    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194616    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194619    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194622    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194628    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194632    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194635    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194638    2580 flags.go:64] FLAG: --port="10250"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194641    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194644    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04e05df61acb2b954"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194648    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194651    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194655    2580 flags.go:64] FLAG: --register-node="true"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194658    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194662    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194665    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194668    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194671    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194674    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194677    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 18:15:19.202288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194681    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194684    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194687    2580 flags.go:64] FLAG: --runonce="false"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194689    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194693    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194696    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194699    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194702    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194705    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194708    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194712    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194714    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194718    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194721    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194724    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194727    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194730    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194734    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194739    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194742    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194745    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194753    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194757    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194760    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194765    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 18:15:19.202926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194768    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194771    2580 flags.go:64] FLAG: --v="2"
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194775    2580 flags.go:64] FLAG: --version="false"
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194779    2580 flags.go:64] FLAG: --vmodule=""
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194783    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.194786    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194907    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194911    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194914    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194918    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194921    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194924    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194926    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194929    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194932    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194935    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194937    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194942    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194945    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194949    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:19.203573 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194953    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194956    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194960    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194963    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194966    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194969    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194971    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194974    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194976    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194981    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194985    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194990    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194993    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194996    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.194999    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195002    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195004    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195007    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195009    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195012    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:19.204058 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195015    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195017    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195020    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195022    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195025    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195027    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195030    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195033    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195035    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195038    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195041    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195043    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195046    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195048    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195051    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195054    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195056    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195059    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195062    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195064    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:19.204638 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195067    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195072    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195074    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195078    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195081    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195084    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195086    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195089    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195092    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195095    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195097    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195100    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195102    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195105    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195108    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195111    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195113    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195116    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195119    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195122    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:19.205449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195125    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195127    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195130    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195132    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195135    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195137    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195140    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195143    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195145    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195148    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195151    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195153    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195156    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195160    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195162    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195166    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195169    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195171    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195174    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195177    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:19.206168 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195179    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195182    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195184    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195187    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.195190    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.195208    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.202426    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.202445    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.202459    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.202466    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202528    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202533    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202537    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202540    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202543    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:19.206688 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202546    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202549    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202552    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202555    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202557    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202561    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202564    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202568    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202571    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202573    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202576    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202579    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202582    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202585    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202587    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202590    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202593    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202596    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202599    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202601    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:19.207068 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202604    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202607    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202609    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202613    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202616    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202620    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202622    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202625    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202628    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202631    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202634    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202637    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202639    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202641    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202644    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202646    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202649    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202652    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202654    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202657    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:19.207626 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202661    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202664    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202666    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202669    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202672    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202674    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202677    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202679    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202683    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202687    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202691    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202695    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202698    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202701    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202703    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202706    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202709    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202713    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202717    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:19.208135 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202720    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202723    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202726    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202729    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202732    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202735    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202738    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202740    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202743    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202746    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202748    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202751    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202754    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202756    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202760    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202763    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202766    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202768    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202771    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202773    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:19.208614 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202776    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202779    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202782    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202784    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202787    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202791    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202795    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202798    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202800    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202803    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202806    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202809    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202811    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202814    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:19.209151 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202817    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.202822    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202943    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202949    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202952    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202955    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202959    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202962    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202964    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202967    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202970    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202973    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202975    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202978    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202981    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:19.209518 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202984    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202986    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202989    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202992    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202994    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.202997    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203000    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203002    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203005    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203008    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203010    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203013    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203016    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203019    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203021    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203024    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203027    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203029    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203032    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203035    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:19.209906 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203037    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203040    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203043    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203045    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203048    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203050    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203053    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203056    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203058    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203061    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203065    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203070    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203073    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203077    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203080    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203083    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203087    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203090    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203093    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:19.210407 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203096    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203099    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203101    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203104    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203106    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203109    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203112    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203114    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203117    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203120    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203122    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203125    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203127    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203130    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203132    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203135    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203138    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203140    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203143    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203145    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:19.210867 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203148    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203151    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203153    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203156    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203158    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203161    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203164    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203166    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203169    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203172    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203175    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203177    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203180    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203182    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203185    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203188    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203190    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203193    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203212    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203216    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203221    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:19.211470 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203224    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203228    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203232    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203235    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203238    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:19.203241    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.203246    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.203374    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.207785    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.208764    2580 server.go:1019] "Starting client certificate rotation"
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.208866    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.208909    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:15:19.211977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.210998    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:15:19.212301 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.212220    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:15:19.222267 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.222245    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 11 18:15:19.226333 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.226314    2580 log.go:25] "Validated CRI v1 image API"
Sep 11 18:15:19.227260 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.227244    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 18:15:19.230416 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.230392    2580 fs.go:135] Filesystem UUIDs: map[00a6ca35-9ff7-4a88-88fe-f28ca3e9993b:/dev/nvme0n1p4 165ecbc5-72ef-46ba-a8ae-dfac60565a5e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 11 18:15:19.230506 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.230414    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 18:15:19.235689 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.235569    2580 manager.go:216] Machine: {Timestamp:2026-09-11 18:15:19.2343824 +0000 UTC m=+0.330976022 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094395 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20ddff26d3ad50a8db96f43aec6340 SystemUUID:ec20ddff-26d3-ad50-a8db-96f43aec6340 BootID:01fd36a6-9ef9-40bb-bddd-d7a0941afd46 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:5f:52:c7:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:5f:52:c7:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:0b:27:6e:02:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 18:15:19.235689 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.235686    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 18:15:19.235806 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.235791    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 18:15:19.239497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.239460    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 18:15:19.239646 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.239501    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-7.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 18:15:19.240855 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.240840    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 18:15:19.240855 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.240857    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 18:15:19.240931 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.240865    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:15:19.241536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.241523    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:15:19.242300 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.242289    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 18:15:19.245884 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.245867    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:15:19.246017 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.246006    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 18:15:19.246049 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.246040    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 18:15:19.246081 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.246050    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 18:15:19.246081 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.246065    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 18:15:19.246081 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.246074    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 11 18:15:19.246165 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.246084    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 18:15:19.246589 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.246567    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:15:19.246654 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.246596    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:15:19.247182 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.247148    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:15:19.250295 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.250224    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 18:15:19.252392 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.252375    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 18:15:19.252457 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.252408    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 18:15:19.254687 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254670    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254697    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254704    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254710    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254719    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254725    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254739    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254746    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254754    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254760    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254769    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 18:15:19.254772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.254775    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 18:15:19.255882 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.255872    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 18:15:19.255882 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.255882    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 18:15:19.258896 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.258876    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-7.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 18:15:19.259549 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.259515    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 18:15:19.259549 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.259536    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-7.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 18:15:19.259858 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.259845    2580 server.go:1268] "Started kubelet"
Sep 11 18:15:19.259958 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.259922    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 18:15:19.259991 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.259937    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 18:15:19.260026 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.260002    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 18:15:19.261055 ip-10-0-129-7 systemd[1]: Started Kubernetes Kubelet.
Sep 11 18:15:19.261266 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.261060    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 18:15:19.261491 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.261474    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 18:15:19.264398 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.264295    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 18:15:19.266310 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.265282    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-7.ec2.internal.18d456eafe2fae30  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-7.ec2.internal,UID:ip-10-0-129-7.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-7.ec2.internal,},FirstTimestamp:2026-09-11 18:15:19.259803184 +0000 UTC m=+0.356396805,LastTimestamp:2026-09-11 18:15:19.259803184 +0000 UTC m=+0.356396805,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-7.ec2.internal,}"
Sep 11 18:15:19.267148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.267127    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 18:15:19.267148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.267133    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 18:15:19.267290 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.267253    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:19.267824 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.267788    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 18:15:19.267969 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.267949    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 18:15:19.268033 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.267974    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 18:15:19.268949 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.268931    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 18:15:19.268949 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.268950    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 18:15:19.269081 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.268950    2580 factory.go:55] Registering systemd factory
Sep 11 18:15:19.269081 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.268973    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 11 18:15:19.269172 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.269089    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t2cgd"
Sep 11 18:15:19.269286 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.269268    2580 factory.go:153] Registering CRI-O factory
Sep 11 18:15:19.269386 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.269301    2580 factory.go:223] Registration of the crio container factory successfully
Sep 11 18:15:19.269619 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.269436    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 18:15:19.269619 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.269477    2580 factory.go:103] Registering Raw factory
Sep 11 18:15:19.269619 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.269504    2580 manager.go:1196] Started watching for new ooms in manager
Sep 11 18:15:19.270149 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.270133    2580 manager.go:349] Starting recovery of all containers
Sep 11 18:15:19.271012 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.270985    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-7.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 18:15:19.271106 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.271082    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 18:15:19.283426 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.283395    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t2cgd"
Sep 11 18:15:19.284179 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.284160    2580 manager.go:354] Recovery completed
Sep 11 18:15:19.291174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.291153    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:19.293618 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.293527    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:19.293618 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.293562    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:19.293618 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.293576    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:19.294267 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.294252    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 18:15:19.294267 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.294261    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:19.294267 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.294264    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 18:15:19.294427 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.294281    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:15:19.295043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.295027    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:19.295121 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.295054    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:19.295121 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.295064    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:19.295955 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.295857    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-7.ec2.internal.18d456eb00328636  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-7.ec2.internal,UID:ip-10-0-129-7.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-7.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-7.ec2.internal,},FirstTimestamp:2026-09-11 18:15:19.29354399 +0000 UTC m=+0.390137613,LastTimestamp:2026-09-11 18:15:19.29354399 +0000 UTC m=+0.390137613,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-7.ec2.internal,}"
Sep 11 18:15:19.296156 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.296142    2580 policy_none.go:49] "None policy: Start"
Sep 11 18:15:19.296233 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.296161    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 18:15:19.296233 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.296172    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 18:15:19.298045 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.298032    2580 policy_none.go:47] "Start"
Sep 11 18:15:19.345342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.343016    2580 manager.go:337] "Starting Device Plugin manager"
Sep 11 18:15:19.345342 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.343071    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 18:15:19.345342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.343085    2580 server.go:87] "Starting device plugin registration server"
Sep 11 18:15:19.345342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.343435    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 18:15:19.345342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.343447    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 18:15:19.345342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.343570    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 18:15:19.345342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.343664    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 18:15:19.345342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.343673    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 18:15:19.345342 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.344397    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 18:15:19.345342 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.344448    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:19.387544 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.387494    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 18:15:19.388796 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.388777    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 18:15:19.388922 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.388833    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 18:15:19.388922 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.388856    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 18:15:19.389008 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.388953    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 18:15:19.391878 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.391851    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:19.444354 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.444122    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:19.445405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.445385    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:19.445501 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.445419    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:19.445501 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.445433    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:19.445501 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.445464    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.454447 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.454425    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.454528 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.454457    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-7.ec2.internal\": node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:19.485427 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.485398    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:19.489435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.489416    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal"]
Sep 11 18:15:19.489540 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.489520    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:19.490518 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.490498    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:19.490610 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.490530    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:19.490610 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.490539    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:19.492767 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.492751    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:19.492873 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.492856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.492917 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.492907    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:19.493638 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.493621    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:19.493761 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.493650    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:19.493761 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.493660    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:19.493761 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.493625    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:19.493761 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.493727    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:19.493761 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.493745    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:19.495948 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.495930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.496030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.495963    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:19.496833 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.496818    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:19.496923 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.496842    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:19.496923 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.496851    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:19.520131 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.520105    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-7.ec2.internal\" not found" node="ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.524625 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.524606    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-7.ec2.internal\" not found" node="ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.570461 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.570427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d40969adc2806df4839ecaab1c9a6cc7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-7.ec2.internal\" (UID: \"d40969adc2806df4839ecaab1c9a6cc7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.570461 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.570460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b95267d9ff34f8b93a90d9a0bb258b60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal\" (UID: \"b95267d9ff34f8b93a90d9a0bb258b60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.570656 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.570479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b95267d9ff34f8b93a90d9a0bb258b60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal\" (UID: \"b95267d9ff34f8b93a90d9a0bb258b60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.585751 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.585718    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:19.671219 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.671165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d40969adc2806df4839ecaab1c9a6cc7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-7.ec2.internal\" (UID: \"d40969adc2806df4839ecaab1c9a6cc7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.671394 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.671224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b95267d9ff34f8b93a90d9a0bb258b60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal\" (UID: \"b95267d9ff34f8b93a90d9a0bb258b60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.671394 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.671251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b95267d9ff34f8b93a90d9a0bb258b60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal\" (UID: \"b95267d9ff34f8b93a90d9a0bb258b60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.671394 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.671258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d40969adc2806df4839ecaab1c9a6cc7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-7.ec2.internal\" (UID: \"d40969adc2806df4839ecaab1c9a6cc7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.671394 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.671285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b95267d9ff34f8b93a90d9a0bb258b60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal\" (UID: \"b95267d9ff34f8b93a90d9a0bb258b60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.671394 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.671322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b95267d9ff34f8b93a90d9a0bb258b60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal\" (UID: \"b95267d9ff34f8b93a90d9a0bb258b60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.686318 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.686285    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:19.787267 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.787168    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:19.823374 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.823334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.827063 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:19.827040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:19.888144 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.888094    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:19.988620 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:19.988587    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:20.080086 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.079974    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:20.088690 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:20.088654    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:20.189217 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:20.189166    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:20.208363 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.208332    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 18:15:20.208531 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.208509    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 18:15:20.208572 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.208508    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 18:15:20.289633 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:20.289589    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:20.290747 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.290720    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 18:10:19 +0000 UTC" deadline="2028-05-20 07:26:27.42870032 +0000 UTC"
Sep 11 18:15:20.290817 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.290749    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14797h11m7.137955973s"
Sep 11 18:15:20.390435 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:20.390361    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:20.428853 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:20.428805    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95267d9ff34f8b93a90d9a0bb258b60.slice/crio-f71ad713041b8b44f57e738287b82ac145d7a48ef3e76eab862da7333659b5db WatchSource:0}: Error finding container f71ad713041b8b44f57e738287b82ac145d7a48ef3e76eab862da7333659b5db: Status 404 returned error can't find the container with id f71ad713041b8b44f57e738287b82ac145d7a48ef3e76eab862da7333659b5db
Sep 11 18:15:20.429402 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:20.429381    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40969adc2806df4839ecaab1c9a6cc7.slice/crio-2d09dfce479d39b29ad1e08831381b9a605c801c4d090c671d76c36f45708114 WatchSource:0}: Error finding container 2d09dfce479d39b29ad1e08831381b9a605c801c4d090c671d76c36f45708114: Status 404 returned error can't find the container with id 2d09dfce479d39b29ad1e08831381b9a605c801c4d090c671d76c36f45708114
Sep 11 18:15:20.433820 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.433801    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:15:20.490670 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:20.490629    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:20.591212 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:20.591180    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:20.691837 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:20.691749    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-7.ec2.internal\" not found"
Sep 11 18:15:20.786663 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.786638    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:20.824341 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.824300    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:20.868322 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.868290    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:20.882660 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.882627    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:15:20.884007 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.883985    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal"
Sep 11 18:15:20.894611 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:20.894581    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:15:21.247039 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.247001    2580 apiserver.go:52] "Watching apiserver"
Sep 11 18:15:21.254146 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.254089    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 18:15:21.254510 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.254484    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-jldjw","openshift-ovn-kubernetes/ovnkube-node-lrh4p","kube-system/global-pull-secret-syncer-8s7n8","kube-system/konnectivity-agent-9zqsb","kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal","openshift-multus/multus-additional-cni-plugins-2lrgh","openshift-multus/network-metrics-daemon-plhmr","openshift-network-diagnostics/network-check-target-jhdmg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h","openshift-cluster-node-tuning-operator/tuned-pdmwk","openshift-dns/node-resolver-cqq8n","openshift-image-registry/node-ca-nrdkw","openshift-multus/multus-dppvt"]
Sep 11 18:15:21.257092 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.257060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.260335 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.259322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.260335 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.259548    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9dl6p\""
Sep 11 18:15:21.260335 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.259800    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 18:15:21.260335 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.259997    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 18:15:21.260335 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.260011    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 18:15:21.260335 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.260184    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 18:15:21.260335 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.260333    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 18:15:21.261876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.261521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:21.261876 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.261607    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:21.261876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.261726    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:21.262581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.262087    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:21.262581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.262395    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fwv28\""
Sep 11 18:15:21.264369 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.263934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:21.266480 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.266460    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 18:15:21.266834 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.266736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mnd8f\""
Sep 11 18:15:21.266976 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.266962    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 18:15:21.267562 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.267402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.268098 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.268036    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 18:15:21.269292 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.269271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-927s8\""
Sep 11 18:15:21.269688 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.269667    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 18:15:21.269882 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.269866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:21.269959 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.269941    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:21.272255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.272134    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:21.272255 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.272219    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:21.275123 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.274979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.277027 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.277006    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.277400 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.277169    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 18:15:21.277400 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.277193    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vm7kf\""
Sep 11 18:15:21.277400 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.277284    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 18:15:21.277634 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.277612    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 18:15:21.279566 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.279547    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.279677 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.279577    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 18:15:21.279940 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.279917    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 18:15:21.280041 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.279925    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 18:15:21.280041 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.280031    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-znxc9\""
Sep 11 18:15:21.280725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.280705    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 18:15:21.280725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.280716    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 18:15:21.280840 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.280799    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 18:15:21.281834 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.281818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.282059 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282039    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 18:15:21.282294 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:21.282372 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282307    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 18:15:21.282372 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-run-k8s-cni-cncf-io\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.282372 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-etc-kubernetes\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.282626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282380    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v54qz\""
Sep 11 18:15:21.282626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6d9\" (UniqueName: \"kubernetes.io/projected/c8f6ac10-f51e-453c-8d72-09cfa833f696-kube-api-access-gt6d9\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.282626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-run\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.282626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-var-lib-kubelet\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.282626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a09a0852-c203-4a28-95cf-ca2e066741b9-tmp\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.282626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1649793f-bd76-49a7-9b4e-61473e8857b2-dbus\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:21.282626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282579    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 18:15:21.282626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-cnibin\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.282626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-run-multus-certs\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8f6ac10-f51e-453c-8d72-09cfa833f696-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8f6ac10-f51e-453c-8d72-09cfa833f696-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-modprobe-d\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-lib-modules\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-host\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fca03eb-a3d5-4625-b302-194c27bf4592-agent-certs\") pod \"konnectivity-agent-9zqsb\" (UID: \"8fca03eb-a3d5-4625-b302-194c27bf4592\") " pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-systemd\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-hostroot\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw7nj\" (UniqueName: \"kubernetes.io/projected/a09a0852-c203-4a28-95cf-ca2e066741b9-kube-api-access-tw7nj\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282927    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.282980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8f6ac10-f51e-453c-8d72-09cfa833f696-cnibin\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.283023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-cni-dir\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.283612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-daemon-config\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.283612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cstr\" (UniqueName: \"kubernetes.io/projected/8fe6b25b-33ee-4180-93f7-e486184b0bfe-kube-api-access-4cstr\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.283612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8f6ac10-f51e-453c-8d72-09cfa833f696-system-cni-dir\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.283755 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fe6b25b-33ee-4180-93f7-e486184b0bfe-cni-binary-copy\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.283755 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-run-netns\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.283854 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-var-lib-cni-bin\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.283905 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-var-lib-kubelet\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.283957 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn48p\" (UniqueName: \"kubernetes.io/projected/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-kube-api-access-mn48p\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:21.283957 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8f6ac10-f51e-453c-8d72-09cfa833f696-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.284077 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.283974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-sysconfig\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.285760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.284741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-sysctl-d\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.285760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.284783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1649793f-bd76-49a7-9b4e-61473e8857b2-kubelet-config\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:21.285760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.284823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fca03eb-a3d5-4625-b302-194c27bf4592-konnectivity-ca\") pod \"konnectivity-agent-9zqsb\" (UID: \"8fca03eb-a3d5-4625-b302-194c27bf4592\") " pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:21.285760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.284860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-socket-dir-parent\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.285760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.284898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:21.285760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.284939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8f6ac10-f51e-453c-8d72-09cfa833f696-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.285760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.284966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-kubernetes\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.285760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.284995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-sysctl-conf\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.286187 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.285773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-tuned\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.286187 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.285899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-system-cni-dir\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.286187 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.285955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-var-lib-cni-multus\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.286187 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.285980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-conf-dir\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.286187 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.286034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-sys\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.286429 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.286329    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:21.286866 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.286845    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 18:15:21.286957 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.286907    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kqztf\""
Sep 11 18:15:21.287227 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.287189    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:21.288781 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.288313    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.290912 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.290889    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 18:15:21.291049 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.291021    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9rcc8\""
Sep 11 18:15:21.291139 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.291125    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 18:15:21.337729 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.337660    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5lp2g"
Sep 11 18:15:21.347290 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.347248    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5lp2g"
Sep 11 18:15:21.368928 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.368898    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 18:15:21.386655 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-sysctl-d\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.386825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2959276-2bdb-41ee-8c15-779298446e02-env-overrides\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.386825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2959276-2bdb-41ee-8c15-779298446e02-ovn-node-metrics-cert\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.386825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1649793f-bd76-49a7-9b4e-61473e8857b2-kubelet-config\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:21.386825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fca03eb-a3d5-4625-b302-194c27bf4592-konnectivity-ca\") pod \"konnectivity-agent-9zqsb\" (UID: \"8fca03eb-a3d5-4625-b302-194c27bf4592\") " pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:21.386825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:21.386825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8f6ac10-f51e-453c-8d72-09cfa833f696-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.386825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-kubernetes\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0a70781-97db-4f2d-9d11-d13d6cfdd56d-iptables-alerter-script\") pod \"iptables-alerter-jldjw\" (UID: \"f0a70781-97db-4f2d-9d11-d13d6cfdd56d\") " pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0a70781-97db-4f2d-9d11-d13d6cfdd56d-host-slash\") pod \"iptables-alerter-jldjw\" (UID: \"f0a70781-97db-4f2d-9d11-d13d6cfdd56d\") " pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-sysctl-d\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-var-lib-cni-multus\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-conf-dir\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-sys\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-device-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.386986    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.386991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-sys-fs\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/355ecf34-6283-43ff-9bbc-1313be1dea3a-hosts-file\") pod \"node-resolver-cqq8n\" (UID: \"355ecf34-6283-43ff-9bbc-1313be1dea3a\") " pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adbddaee-40dd-465e-bd6c-7a32d384346a-host\") pod \"node-ca-nrdkw\" (UID: \"adbddaee-40dd-465e-bd6c-7a32d384346a\") " pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.387077    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs podName:cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:21.887039056 +0000 UTC m=+2.983632665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs") pod "network-metrics-daemon-plhmr" (UID: "cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:21.387174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1649793f-bd76-49a7-9b4e-61473e8857b2-kubelet-config\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:21.387857 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fca03eb-a3d5-4625-b302-194c27bf4592-konnectivity-ca\") pod \"konnectivity-agent-9zqsb\" (UID: \"8fca03eb-a3d5-4625-b302-194c27bf4592\") " pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:21.387857 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-conf-dir\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.387857 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-run-openvswitch\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.387857 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-var-lib-cni-multus\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.387857 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-kubernetes\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.387857 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6d9\" (UniqueName: \"kubernetes.io/projected/c8f6ac10-f51e-453c-8d72-09cfa833f696-kube-api-access-gt6d9\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.388112 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-sys\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.388112 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-run\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.388112 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7vt\" (UniqueName: \"kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt\") pod \"network-check-target-jhdmg\" (UID: \"c5329603-85e2-4ad4-a14e-4b37e0b1c105\") " pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:21.388112 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-slash\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.388112 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.387984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-run\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.388112 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-run-systemd\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.388112 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-var-lib-openvswitch\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-cnibin\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-run-multus-certs\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-run-multus-certs\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-cnibin\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8f6ac10-f51e-453c-8d72-09cfa833f696-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-lib-modules\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355ecf34-6283-43ff-9bbc-1313be1dea3a-tmp-dir\") pod \"node-resolver-cqq8n\" (UID: \"355ecf34-6283-43ff-9bbc-1313be1dea3a\") " pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-systemd\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8f6ac10-f51e-453c-8d72-09cfa833f696-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtchb\" (UniqueName: \"kubernetes.io/projected/adbddaee-40dd-465e-bd6c-7a32d384346a-kube-api-access-qtchb\") pod \"node-ca-nrdkw\" (UID: \"adbddaee-40dd-465e-bd6c-7a32d384346a\") " pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-run-netns\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.388435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-log-socket\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-hostroot\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-lib-modules\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw7nj\" (UniqueName: \"kubernetes.io/projected/a09a0852-c203-4a28-95cf-ca2e066741b9-kube-api-access-tw7nj\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-etc-selinux\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-systemd\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-etc-openvswitch\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-cni-netd\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2959276-2bdb-41ee-8c15-779298446e02-ovnkube-script-lib\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-var-lib-kubelet\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn48p\" (UniqueName: \"kubernetes.io/projected/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-kube-api-access-mn48p\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-sysconfig\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-socket-dir-parent\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-tuned\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8f6ac10-f51e-453c-8d72-09cfa833f696-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.388975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmtlc\" (UniqueName: \"kubernetes.io/projected/f0a70781-97db-4f2d-9d11-d13d6cfdd56d-kube-api-access-nmtlc\") pod \"iptables-alerter-jldjw\" (UID: \"f0a70781-97db-4f2d-9d11-d13d6cfdd56d\") " pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-hostroot\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-system-cni-dir\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-socket-dir-parent\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-kubelet\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-sysconfig\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-node-log\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvpwh\" (UniqueName: \"kubernetes.io/projected/a2959276-2bdb-41ee-8c15-779298446e02-kube-api-access-vvpwh\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-system-cni-dir\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.388986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-var-lib-kubelet\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-run-k8s-cni-cncf-io\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.389128    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-etc-kubernetes\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-etc-kubernetes\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-var-lib-kubelet\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a09a0852-c203-4a28-95cf-ca2e066741b9-tmp\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389228    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 18:15:21.389728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2959276-2bdb-41ee-8c15-779298446e02-ovnkube-config\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1649793f-bd76-49a7-9b4e-61473e8857b2-dbus\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8f6ac10-f51e-453c-8d72-09cfa833f696-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.389324    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret podName:1649793f-bd76-49a7-9b4e-61473e8857b2 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:21.889307385 +0000 UTC m=+2.985901009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret") pod "global-pull-secret-syncer-8s7n8" (UID: "1649793f-bd76-49a7-9b4e-61473e8857b2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-modprobe-d\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-run-k8s-cni-cncf-io\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-host\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-registration-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-systemd-units\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fca03eb-a3d5-4625-b302-194c27bf4592-agent-certs\") pod \"konnectivity-agent-9zqsb\" (UID: \"8fca03eb-a3d5-4625-b302-194c27bf4592\") " pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-socket-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmct\" (UniqueName: \"kubernetes.io/projected/d81ade46-ec28-4514-8654-19df644ee51b-kube-api-access-5mmct\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adbddaee-40dd-465e-bd6c-7a32d384346a-serviceca\") pod \"node-ca-nrdkw\" (UID: \"adbddaee-40dd-465e-bd6c-7a32d384346a\") " pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-run-ovn\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1649793f-bd76-49a7-9b4e-61473e8857b2-dbus\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-host\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.390536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-modprobe-d\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-var-lib-kubelet\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2sb\" (UniqueName: \"kubernetes.io/projected/355ecf34-6283-43ff-9bbc-1313be1dea3a-kube-api-access-gq2sb\") pod \"node-resolver-cqq8n\" (UID: \"355ecf34-6283-43ff-9bbc-1313be1dea3a\") " pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8f6ac10-f51e-453c-8d72-09cfa833f696-cnibin\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8f6ac10-f51e-453c-8d72-09cfa833f696-cnibin\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-cni-dir\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8f6ac10-f51e-453c-8d72-09cfa833f696-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-daemon-config\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cstr\" (UniqueName: \"kubernetes.io/projected/8fe6b25b-33ee-4180-93f7-e486184b0bfe-kube-api-access-4cstr\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.389986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-cni-dir\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8f6ac10-f51e-453c-8d72-09cfa833f696-system-cni-dir\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-run-ovn-kubernetes\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-cni-bin\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-sysctl-conf\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fe6b25b-33ee-4180-93f7-e486184b0bfe-cni-binary-copy\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-run-netns\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-var-lib-cni-bin\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.391220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8f6ac10-f51e-453c-8d72-09cfa833f696-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.392055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8f6ac10-f51e-453c-8d72-09cfa833f696-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.392055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-run-netns\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.392055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fe6b25b-33ee-4180-93f7-e486184b0bfe-multus-daemon-config\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.392055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8f6ac10-f51e-453c-8d72-09cfa833f696-system-cni-dir\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.392055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-sysctl-conf\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.392055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.390700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fe6b25b-33ee-4180-93f7-e486184b0bfe-host-var-lib-cni-bin\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.392055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.391089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fe6b25b-33ee-4180-93f7-e486184b0bfe-cni-binary-copy\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.393263 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.393162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fca03eb-a3d5-4625-b302-194c27bf4592-agent-certs\") pod \"konnectivity-agent-9zqsb\" (UID: \"8fca03eb-a3d5-4625-b302-194c27bf4592\") " pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:21.393380 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.393337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a09a0852-c203-4a28-95cf-ca2e066741b9-etc-tuned\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.393380 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.393373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a09a0852-c203-4a28-95cf-ca2e066741b9-tmp\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.395104 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.395022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal" event={"ID":"b95267d9ff34f8b93a90d9a0bb258b60","Type":"ContainerStarted","Data":"f71ad713041b8b44f57e738287b82ac145d7a48ef3e76eab862da7333659b5db"}
Sep 11 18:15:21.397637 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.397548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal" event={"ID":"d40969adc2806df4839ecaab1c9a6cc7","Type":"ContainerStarted","Data":"2d09dfce479d39b29ad1e08831381b9a605c801c4d090c671d76c36f45708114"}
Sep 11 18:15:21.400617 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.400540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw7nj\" (UniqueName: \"kubernetes.io/projected/a09a0852-c203-4a28-95cf-ca2e066741b9-kube-api-access-tw7nj\") pod \"tuned-pdmwk\" (UID: \"a09a0852-c203-4a28-95cf-ca2e066741b9\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.402051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.402026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cstr\" (UniqueName: \"kubernetes.io/projected/8fe6b25b-33ee-4180-93f7-e486184b0bfe-kube-api-access-4cstr\") pod \"multus-dppvt\" (UID: \"8fe6b25b-33ee-4180-93f7-e486184b0bfe\") " pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.402352 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.402328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6d9\" (UniqueName: \"kubernetes.io/projected/c8f6ac10-f51e-453c-8d72-09cfa833f696-kube-api-access-gt6d9\") pod \"multus-additional-cni-plugins-2lrgh\" (UID: \"c8f6ac10-f51e-453c-8d72-09cfa833f696\") " pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.402795 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.402768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn48p\" (UniqueName: \"kubernetes.io/projected/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-kube-api-access-mn48p\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:21.451810 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.451769    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:21.491282 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-run-ovn-kubernetes\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491282 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-cni-bin\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-cni-bin\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2959276-2bdb-41ee-8c15-779298446e02-env-overrides\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-run-ovn-kubernetes\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2959276-2bdb-41ee-8c15-779298446e02-ovn-node-metrics-cert\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0a70781-97db-4f2d-9d11-d13d6cfdd56d-iptables-alerter-script\") pod \"iptables-alerter-jldjw\" (UID: \"f0a70781-97db-4f2d-9d11-d13d6cfdd56d\") " pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.491537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0a70781-97db-4f2d-9d11-d13d6cfdd56d-host-slash\") pod \"iptables-alerter-jldjw\" (UID: \"f0a70781-97db-4f2d-9d11-d13d6cfdd56d\") " pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.491537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-device-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-sys-fs\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/355ecf34-6283-43ff-9bbc-1313be1dea3a-hosts-file\") pod \"node-resolver-cqq8n\" (UID: \"355ecf34-6283-43ff-9bbc-1313be1dea3a\") " pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adbddaee-40dd-465e-bd6c-7a32d384346a-host\") pod \"node-ca-nrdkw\" (UID: \"adbddaee-40dd-465e-bd6c-7a32d384346a\") " pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0a70781-97db-4f2d-9d11-d13d6cfdd56d-host-slash\") pod \"iptables-alerter-jldjw\" (UID: \"f0a70781-97db-4f2d-9d11-d13d6cfdd56d\") " pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-run-openvswitch\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-device-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-run-openvswitch\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7vt\" (UniqueName: \"kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt\") pod \"network-check-target-jhdmg\" (UID: \"c5329603-85e2-4ad4-a14e-4b37e0b1c105\") " pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-slash\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/355ecf34-6283-43ff-9bbc-1313be1dea3a-hosts-file\") pod \"node-resolver-cqq8n\" (UID: \"355ecf34-6283-43ff-9bbc-1313be1dea3a\") " pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-run-systemd\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-var-lib-openvswitch\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-sys-fs\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355ecf34-6283-43ff-9bbc-1313be1dea3a-tmp-dir\") pod \"node-resolver-cqq8n\" (UID: \"355ecf34-6283-43ff-9bbc-1313be1dea3a\") " pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-slash\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtchb\" (UniqueName: \"kubernetes.io/projected/adbddaee-40dd-465e-bd6c-7a32d384346a-kube-api-access-qtchb\") pod \"node-ca-nrdkw\" (UID: \"adbddaee-40dd-465e-bd6c-7a32d384346a\") " pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-run-netns\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.491876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adbddaee-40dd-465e-bd6c-7a32d384346a-host\") pod \"node-ca-nrdkw\" (UID: \"adbddaee-40dd-465e-bd6c-7a32d384346a\") " pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2959276-2bdb-41ee-8c15-779298446e02-env-overrides\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-run-netns\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-var-lib-openvswitch\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-run-systemd\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-log-socket\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.491982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-etc-selinux\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-log-socket\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-etc-openvswitch\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-cni-netd\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-etc-selinux\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2959276-2bdb-41ee-8c15-779298446e02-ovnkube-script-lib\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0a70781-97db-4f2d-9d11-d13d6cfdd56d-iptables-alerter-script\") pod \"iptables-alerter-jldjw\" (UID: \"f0a70781-97db-4f2d-9d11-d13d6cfdd56d\") " pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-etc-openvswitch\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-cni-netd\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.492725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355ecf34-6283-43ff-9bbc-1313be1dea3a-tmp-dir\") pod \"node-resolver-cqq8n\" (UID: \"355ecf34-6283-43ff-9bbc-1313be1dea3a\") " pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtlc\" (UniqueName: \"kubernetes.io/projected/f0a70781-97db-4f2d-9d11-d13d6cfdd56d-kube-api-access-nmtlc\") pod \"iptables-alerter-jldjw\" (UID: \"f0a70781-97db-4f2d-9d11-d13d6cfdd56d\") " pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-kubelet\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-node-log\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvpwh\" (UniqueName: \"kubernetes.io/projected/a2959276-2bdb-41ee-8c15-779298446e02-kube-api-access-vvpwh\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-host-kubelet\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-node-log\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2959276-2bdb-41ee-8c15-779298446e02-ovnkube-config\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-registration-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-systemd-units\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-socket-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmct\" (UniqueName: \"kubernetes.io/projected/d81ade46-ec28-4514-8654-19df644ee51b-kube-api-access-5mmct\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-registration-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adbddaee-40dd-465e-bd6c-7a32d384346a-serviceca\") pod \"node-ca-nrdkw\" (UID: \"adbddaee-40dd-465e-bd6c-7a32d384346a\") " pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-run-ovn\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2sb\" (UniqueName: \"kubernetes.io/projected/355ecf34-6283-43ff-9bbc-1313be1dea3a-kube-api-access-gq2sb\") pod \"node-resolver-cqq8n\" (UID: \"355ecf34-6283-43ff-9bbc-1313be1dea3a\") " pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.493986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d81ade46-ec28-4514-8654-19df644ee51b-socket-dir\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.493986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-systemd-units\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2959276-2bdb-41ee-8c15-779298446e02-ovnkube-script-lib\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2959276-2bdb-41ee-8c15-779298446e02-run-ovn\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2959276-2bdb-41ee-8c15-779298446e02-ovnkube-config\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.493986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.492959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adbddaee-40dd-465e-bd6c-7a32d384346a-serviceca\") pod \"node-ca-nrdkw\" (UID: \"adbddaee-40dd-465e-bd6c-7a32d384346a\") " pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.495788 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.495741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2959276-2bdb-41ee-8c15-779298446e02-ovn-node-metrics-cert\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.498157 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.498094    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:21.498157 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.498120    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:21.498157 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.498132    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ww7vt for pod openshift-network-diagnostics/network-check-target-jhdmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:21.498360 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.498215    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt podName:c5329603-85e2-4ad4-a14e-4b37e0b1c105 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:21.998180463 +0000 UTC m=+3.094774084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ww7vt" (UniqueName: "kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt") pod "network-check-target-jhdmg" (UID: "c5329603-85e2-4ad4-a14e-4b37e0b1c105") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:21.501992 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.501960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmtlc\" (UniqueName: \"kubernetes.io/projected/f0a70781-97db-4f2d-9d11-d13d6cfdd56d-kube-api-access-nmtlc\") pod \"iptables-alerter-jldjw\" (UID: \"f0a70781-97db-4f2d-9d11-d13d6cfdd56d\") " pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.502112 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.502001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmct\" (UniqueName: \"kubernetes.io/projected/d81ade46-ec28-4514-8654-19df644ee51b-kube-api-access-5mmct\") pod \"aws-ebs-csi-driver-node-hf67h\" (UID: \"d81ade46-ec28-4514-8654-19df644ee51b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.502559 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.502443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtchb\" (UniqueName: \"kubernetes.io/projected/adbddaee-40dd-465e-bd6c-7a32d384346a-kube-api-access-qtchb\") pod \"node-ca-nrdkw\" (UID: \"adbddaee-40dd-465e-bd6c-7a32d384346a\") " pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.502759 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.502736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvpwh\" (UniqueName: \"kubernetes.io/projected/a2959276-2bdb-41ee-8c15-779298446e02-kube-api-access-vvpwh\") pod \"ovnkube-node-lrh4p\" (UID: \"a2959276-2bdb-41ee-8c15-779298446e02\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.504647 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.504620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2sb\" (UniqueName: \"kubernetes.io/projected/355ecf34-6283-43ff-9bbc-1313be1dea3a-kube-api-access-gq2sb\") pod \"node-resolver-cqq8n\" (UID: \"355ecf34-6283-43ff-9bbc-1313be1dea3a\") " pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.569960 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.569923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2lrgh"
Sep 11 18:15:21.578599 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.578567    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pdmwk"
Sep 11 18:15:21.588496 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.588464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:21.595567 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.595531    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dppvt"
Sep 11 18:15:21.604484 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.604446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrdkw"
Sep 11 18:15:21.611574 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.611543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:21.619516 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.619483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h"
Sep 11 18:15:21.628262 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.628230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jldjw"
Sep 11 18:15:21.633990 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.633964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cqq8n"
Sep 11 18:15:21.896590 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.896487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:21.896590 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:21.896556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:21.896790 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.896667    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:21.896790 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.896692    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:21.896790 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.896738    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs podName:cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:22.896724525 +0000 UTC m=+3.993318134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs") pod "network-metrics-daemon-plhmr" (UID: "cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:21.896790 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:21.896757    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret podName:1649793f-bd76-49a7-9b4e-61473e8857b2 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:22.896746453 +0000 UTC m=+3.993340078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret") pod "global-pull-secret-syncer-8s7n8" (UID: "1649793f-bd76-49a7-9b4e-61473e8857b2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:22.097772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.097727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7vt\" (UniqueName: \"kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt\") pod \"network-check-target-jhdmg\" (UID: \"c5329603-85e2-4ad4-a14e-4b37e0b1c105\") " pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:22.097953 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:22.097923    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:22.097953 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:22.097948    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:22.098070 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:22.097961    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ww7vt for pod openshift-network-diagnostics/network-check-target-jhdmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:22.098070 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:22.098030    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt podName:c5329603-85e2-4ad4-a14e-4b37e0b1c105 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:23.098009446 +0000 UTC m=+4.194603074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww7vt" (UniqueName: "kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt") pod "network-check-target-jhdmg" (UID: "c5329603-85e2-4ad4-a14e-4b37e0b1c105") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:22.333883 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:22.333854    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f6ac10_f51e_453c_8d72_09cfa833f696.slice/crio-c8ab2a2b3a4e478ed62b018f88d1f0f700051967c18d05bbbf89402f199212d1 WatchSource:0}: Error finding container c8ab2a2b3a4e478ed62b018f88d1f0f700051967c18d05bbbf89402f199212d1: Status 404 returned error can't find the container with id c8ab2a2b3a4e478ed62b018f88d1f0f700051967c18d05bbbf89402f199212d1
Sep 11 18:15:22.335746 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:22.335722    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09a0852_c203_4a28_95cf_ca2e066741b9.slice/crio-e7316c0364df9a921d0e39797f3c6d8b531c0f199f3f9b8509a8dd120c18451c WatchSource:0}: Error finding container e7316c0364df9a921d0e39797f3c6d8b531c0f199f3f9b8509a8dd120c18451c: Status 404 returned error can't find the container with id e7316c0364df9a921d0e39797f3c6d8b531c0f199f3f9b8509a8dd120c18451c
Sep 11 18:15:22.336909 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:22.336884    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe6b25b_33ee_4180_93f7_e486184b0bfe.slice/crio-5763debc2e5e95f6e548dfecd8932289ab62913682131a298fa9e1cc1eef12a6 WatchSource:0}: Error finding container 5763debc2e5e95f6e548dfecd8932289ab62913682131a298fa9e1cc1eef12a6: Status 404 returned error can't find the container with id 5763debc2e5e95f6e548dfecd8932289ab62913682131a298fa9e1cc1eef12a6
Sep 11 18:15:22.338316 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:22.338291    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fca03eb_a3d5_4625_b302_194c27bf4592.slice/crio-b625bd6362cd7f110e13d3d3b4fc8c4c27d705ad7d941d831aea1a939524c750 WatchSource:0}: Error finding container b625bd6362cd7f110e13d3d3b4fc8c4c27d705ad7d941d831aea1a939524c750: Status 404 returned error can't find the container with id b625bd6362cd7f110e13d3d3b4fc8c4c27d705ad7d941d831aea1a939524c750
Sep 11 18:15:22.339649 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:22.339620    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbddaee_40dd_465e_bd6c_7a32d384346a.slice/crio-4015bfbe3a2e8bcfba5530ac31aa1092caf86b469d85d4bb3892a6d581141c93 WatchSource:0}: Error finding container 4015bfbe3a2e8bcfba5530ac31aa1092caf86b469d85d4bb3892a6d581141c93: Status 404 returned error can't find the container with id 4015bfbe3a2e8bcfba5530ac31aa1092caf86b469d85d4bb3892a6d581141c93
Sep 11 18:15:22.351703 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.351656    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:21 +0000 UTC" deadline="2028-05-09 01:11:18.051946816 +0000 UTC"
Sep 11 18:15:22.351703 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.351695    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14526h55m55.700255776s"
Sep 11 18:15:22.364333 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:22.364307    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81ade46_ec28_4514_8654_19df644ee51b.slice/crio-34dd2b655059538bb630d872bd817179947f7d00f8f4c5251bb4d20e1165a5be WatchSource:0}: Error finding container 34dd2b655059538bb630d872bd817179947f7d00f8f4c5251bb4d20e1165a5be: Status 404 returned error can't find the container with id 34dd2b655059538bb630d872bd817179947f7d00f8f4c5251bb4d20e1165a5be
Sep 11 18:15:22.364724 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:22.364701    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355ecf34_6283_43ff_9bbc_1313be1dea3a.slice/crio-b38e059befb0ab6917ea968b3cf1f792df44e2494ffe8ea060a317d95573d666 WatchSource:0}: Error finding container b38e059befb0ab6917ea968b3cf1f792df44e2494ffe8ea060a317d95573d666: Status 404 returned error can't find the container with id b38e059befb0ab6917ea968b3cf1f792df44e2494ffe8ea060a317d95573d666
Sep 11 18:15:22.366383 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:22.366353    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2959276_2bdb_41ee_8c15_779298446e02.slice/crio-1f2847bdcb4230883c538b3df58635b6bbc3d3bcbd2cafc8b00d57a6cedf4f7d WatchSource:0}: Error finding container 1f2847bdcb4230883c538b3df58635b6bbc3d3bcbd2cafc8b00d57a6cedf4f7d: Status 404 returned error can't find the container with id 1f2847bdcb4230883c538b3df58635b6bbc3d3bcbd2cafc8b00d57a6cedf4f7d
Sep 11 18:15:22.367406 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:22.367379    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a70781_97db_4f2d_9d11_d13d6cfdd56d.slice/crio-a95de147adce64d5677d44bafdf5f0e8ecbf64f7ccf18321fef3d14695514df3 WatchSource:0}: Error finding container a95de147adce64d5677d44bafdf5f0e8ecbf64f7ccf18321fef3d14695514df3: Status 404 returned error can't find the container with id a95de147adce64d5677d44bafdf5f0e8ecbf64f7ccf18321fef3d14695514df3
Sep 11 18:15:22.402005 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.401968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrdkw" event={"ID":"adbddaee-40dd-465e-bd6c-7a32d384346a","Type":"ContainerStarted","Data":"4015bfbe3a2e8bcfba5530ac31aa1092caf86b469d85d4bb3892a6d581141c93"}
Sep 11 18:15:22.404549 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.404518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9zqsb" event={"ID":"8fca03eb-a3d5-4625-b302-194c27bf4592","Type":"ContainerStarted","Data":"b625bd6362cd7f110e13d3d3b4fc8c4c27d705ad7d941d831aea1a939524c750"}
Sep 11 18:15:22.407097 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.407067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pdmwk" event={"ID":"a09a0852-c203-4a28-95cf-ca2e066741b9","Type":"ContainerStarted","Data":"e7316c0364df9a921d0e39797f3c6d8b531c0f199f3f9b8509a8dd120c18451c"}
Sep 11 18:15:22.408808 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.408772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p" event={"ID":"a2959276-2bdb-41ee-8c15-779298446e02","Type":"ContainerStarted","Data":"1f2847bdcb4230883c538b3df58635b6bbc3d3bcbd2cafc8b00d57a6cedf4f7d"}
Sep 11 18:15:22.409986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.409961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dppvt" event={"ID":"8fe6b25b-33ee-4180-93f7-e486184b0bfe","Type":"ContainerStarted","Data":"5763debc2e5e95f6e548dfecd8932289ab62913682131a298fa9e1cc1eef12a6"}
Sep 11 18:15:22.412861 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.412831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lrgh" event={"ID":"c8f6ac10-f51e-453c-8d72-09cfa833f696","Type":"ContainerStarted","Data":"c8ab2a2b3a4e478ed62b018f88d1f0f700051967c18d05bbbf89402f199212d1"}
Sep 11 18:15:22.417119 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.417091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cqq8n" event={"ID":"355ecf34-6283-43ff-9bbc-1313be1dea3a","Type":"ContainerStarted","Data":"b38e059befb0ab6917ea968b3cf1f792df44e2494ffe8ea060a317d95573d666"}
Sep 11 18:15:22.418641 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.418610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jldjw" event={"ID":"f0a70781-97db-4f2d-9d11-d13d6cfdd56d","Type":"ContainerStarted","Data":"a95de147adce64d5677d44bafdf5f0e8ecbf64f7ccf18321fef3d14695514df3"}
Sep 11 18:15:22.419856 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.419833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h" event={"ID":"d81ade46-ec28-4514-8654-19df644ee51b","Type":"ContainerStarted","Data":"34dd2b655059538bb630d872bd817179947f7d00f8f4c5251bb4d20e1165a5be"}
Sep 11 18:15:22.831698 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.831192    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:22.904242 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.903276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:22.904242 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:22.903416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:22.904242 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:22.903587    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:22.904242 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:22.903678    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs podName:cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:24.903656659 +0000 UTC m=+6.000250284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs") pod "network-metrics-daemon-plhmr" (UID: "cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:22.904242 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:22.904090    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:22.904242 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:22.904136    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret podName:1649793f-bd76-49a7-9b4e-61473e8857b2 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:24.904121107 +0000 UTC m=+6.000714730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret") pod "global-pull-secret-syncer-8s7n8" (UID: "1649793f-bd76-49a7-9b4e-61473e8857b2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:23.105276 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:23.105226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7vt\" (UniqueName: \"kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt\") pod \"network-check-target-jhdmg\" (UID: \"c5329603-85e2-4ad4-a14e-4b37e0b1c105\") " pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:23.105450 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:23.105409    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:23.105450 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:23.105432    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:23.105450 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:23.105445    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ww7vt for pod openshift-network-diagnostics/network-check-target-jhdmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:23.105629 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:23.105507    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt podName:c5329603-85e2-4ad4-a14e-4b37e0b1c105 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:25.105487638 +0000 UTC m=+6.202081264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww7vt" (UniqueName: "kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt") pod "network-check-target-jhdmg" (UID: "c5329603-85e2-4ad4-a14e-4b37e0b1c105") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:23.352826 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:23.352650    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:21 +0000 UTC" deadline="2028-05-26 16:14:07.245095484 +0000 UTC"
Sep 11 18:15:23.352826 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:23.352708    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14949h58m43.892391813s"
Sep 11 18:15:23.389441 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:23.389408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:23.389652 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:23.389541    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:23.389953 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:23.389934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:23.390107 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:23.390030    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:23.390107 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:23.390099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:23.390268 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:23.390187    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:23.431154 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:23.431115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal" event={"ID":"b95267d9ff34f8b93a90d9a0bb258b60","Type":"ContainerDied","Data":"e28ff72ee6aa17b534d9e6c46a18a07d0396402534c1182075a1e478d0bd9c0b"}
Sep 11 18:15:23.431348 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:23.430177    2580 generic.go:356] "Generic (PLEG): container finished" podID="b95267d9ff34f8b93a90d9a0bb258b60" containerID="e28ff72ee6aa17b534d9e6c46a18a07d0396402534c1182075a1e478d0bd9c0b" exitCode=0
Sep 11 18:15:23.438933 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:23.438779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal" event={"ID":"d40969adc2806df4839ecaab1c9a6cc7","Type":"ContainerStarted","Data":"31fdd6618085684fa17e730e3891bf756b7d8704517f254c8e4fa3e28aa5f176"}
Sep 11 18:15:24.462072 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:24.462033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal" event={"ID":"b95267d9ff34f8b93a90d9a0bb258b60","Type":"ContainerStarted","Data":"cdd2ee716a7056d7cd1b2a1c88249441bd7dcfca559859e0435c0d685d28945d"}
Sep 11 18:15:24.476990 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:24.476939    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-7.ec2.internal" podStartSLOduration=4.476919411 podStartE2EDuration="4.476919411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:23.461862516 +0000 UTC m=+4.558456148" watchObservedRunningTime="2026-09-11 18:15:24.476919411 +0000 UTC m=+5.573513043"
Sep 11 18:15:24.477164 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:24.477021    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-7.ec2.internal" podStartSLOduration=4.477014462 podStartE2EDuration="4.477014462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:24.476635195 +0000 UTC m=+5.573228826" watchObservedRunningTime="2026-09-11 18:15:24.477014462 +0000 UTC m=+5.573608092"
Sep 11 18:15:24.922688 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:24.922651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:24.922871 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:24.922723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:24.922871 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:24.922861    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:24.922998 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:24.922923    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs podName:cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:28.922905483 +0000 UTC m=+10.019499098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs") pod "network-metrics-daemon-plhmr" (UID: "cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:24.923375 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:24.923329    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:24.923463 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:24.923398    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret podName:1649793f-bd76-49a7-9b4e-61473e8857b2 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:28.923381653 +0000 UTC m=+10.019975277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret") pod "global-pull-secret-syncer-8s7n8" (UID: "1649793f-bd76-49a7-9b4e-61473e8857b2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:25.124744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:25.124708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7vt\" (UniqueName: \"kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt\") pod \"network-check-target-jhdmg\" (UID: \"c5329603-85e2-4ad4-a14e-4b37e0b1c105\") " pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:25.124914 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:25.124902    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:25.124987 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:25.124922    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:25.124987 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:25.124935    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ww7vt for pod openshift-network-diagnostics/network-check-target-jhdmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:25.125084 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:25.124994    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt podName:c5329603-85e2-4ad4-a14e-4b37e0b1c105 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:29.12497534 +0000 UTC m=+10.221568953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww7vt" (UniqueName: "kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt") pod "network-check-target-jhdmg" (UID: "c5329603-85e2-4ad4-a14e-4b37e0b1c105") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:25.389524 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:25.389486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:25.389710 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:25.389613    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:25.390037 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:25.389993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:25.390139 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:25.390096    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:25.390236 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:25.390149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:25.390298 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:25.390248    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:27.394833 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:27.393957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:27.394833 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:27.394079    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:27.394833 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:27.394491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:27.394833 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:27.394576    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:27.394833 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:27.394644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:27.394833 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:27.394726    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:28.959170 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:28.958995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:28.959170 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:28.959098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:28.959170 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:28.959142    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:28.959838 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:28.959226    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:28.959838 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:28.959235    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs podName:cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:36.959213558 +0000 UTC m=+18.055807194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs") pod "network-metrics-daemon-plhmr" (UID: "cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:28.959838 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:28.959277    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret podName:1649793f-bd76-49a7-9b4e-61473e8857b2 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:36.959261173 +0000 UTC m=+18.055854797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret") pod "global-pull-secret-syncer-8s7n8" (UID: "1649793f-bd76-49a7-9b4e-61473e8857b2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:29.160569 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:29.160525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7vt\" (UniqueName: \"kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt\") pod \"network-check-target-jhdmg\" (UID: \"c5329603-85e2-4ad4-a14e-4b37e0b1c105\") " pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:29.160814 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:29.160746    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:29.160814 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:29.160767    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:29.160814 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:29.160780    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ww7vt for pod openshift-network-diagnostics/network-check-target-jhdmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:29.160999 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:29.160843    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt podName:c5329603-85e2-4ad4-a14e-4b37e0b1c105 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:37.16082483 +0000 UTC m=+18.257418462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww7vt" (UniqueName: "kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt") pod "network-check-target-jhdmg" (UID: "c5329603-85e2-4ad4-a14e-4b37e0b1c105") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:29.393361 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:29.393326    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:29.393361 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:29.393360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:29.393579 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:29.393333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:29.393579 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:29.393466    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:29.393579 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:29.393551    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:29.393753 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:29.393627    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:31.390127 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:31.390085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:31.390127 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:31.390109    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:31.390834 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:31.390085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:31.390834 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:31.390239    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:31.390834 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:31.390313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:31.390834 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:31.390410    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:33.389861 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:33.389807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:33.390423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:33.389807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:33.390423 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:33.389947    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:33.390423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:33.389981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:33.390423 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:33.390093    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:33.390423 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:33.390174    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:35.389337 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:35.389299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:35.389929 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:35.389299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:35.389929 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:35.389432    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:35.389929 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:35.389305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:35.389929 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:35.389522    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:35.389929 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:35.389625    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:37.020332 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:37.020289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:37.020888 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:37.020360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:37.020888 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.020473    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:37.020888 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.020554    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret podName:1649793f-bd76-49a7-9b4e-61473e8857b2 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:53.020533481 +0000 UTC m=+34.117127114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret") pod "global-pull-secret-syncer-8s7n8" (UID: "1649793f-bd76-49a7-9b4e-61473e8857b2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:37.020888 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.020473    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:37.020888 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.020616    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs podName:cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:53.020602197 +0000 UTC m=+34.117195815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs") pod "network-metrics-daemon-plhmr" (UID: "cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:37.221466 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:37.221430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7vt\" (UniqueName: \"kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt\") pod \"network-check-target-jhdmg\" (UID: \"c5329603-85e2-4ad4-a14e-4b37e0b1c105\") " pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:37.221645 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.221584    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:37.221645 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.221603    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:37.221645 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.221616    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ww7vt for pod openshift-network-diagnostics/network-check-target-jhdmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:37.221761 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.221680    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt podName:c5329603-85e2-4ad4-a14e-4b37e0b1c105 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:53.221663186 +0000 UTC m=+34.318256821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww7vt" (UniqueName: "kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt") pod "network-check-target-jhdmg" (UID: "c5329603-85e2-4ad4-a14e-4b37e0b1c105") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:37.389569 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:37.389534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:37.389746 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:37.389546    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:37.389746 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:37.389660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:37.389746 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.389663    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:37.389930 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.389780    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:37.389930 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:37.389882    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:39.390648 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:39.390600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:39.391071 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:39.390695    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:39.391071 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:39.390713    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:39.391071 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:39.390825    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:39.391071 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:39.390884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:39.391071 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:39.390955    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:40.497271 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.497009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrdkw" event={"ID":"adbddaee-40dd-465e-bd6c-7a32d384346a","Type":"ContainerStarted","Data":"9d4175556e8cc71640d280adbd2c894fc38e5c6c3da91a631925834291da8b97"}
Sep 11 18:15:40.498586 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.498551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9zqsb" event={"ID":"8fca03eb-a3d5-4625-b302-194c27bf4592","Type":"ContainerStarted","Data":"a8df3dbf508c22580068a5373f59b766dd37cd8fa9f8aaab11d75524133d285a"}
Sep 11 18:15:40.500611 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.500226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pdmwk" event={"ID":"a09a0852-c203-4a28-95cf-ca2e066741b9","Type":"ContainerStarted","Data":"5c024d2bae1074242aafbb237d39a463fa97de5b35e49c3d68a2b2034245aa2c"}
Sep 11 18:15:40.503251 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.503218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p" event={"ID":"a2959276-2bdb-41ee-8c15-779298446e02","Type":"ContainerStarted","Data":"1a43626e84305bc7777501bab7b5f5a736b76717da1a23e2d33e419677751e9f"}
Sep 11 18:15:40.503361 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.503312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p" event={"ID":"a2959276-2bdb-41ee-8c15-779298446e02","Type":"ContainerStarted","Data":"d901c4fe613f4e5fc6cbf50cca5ec9ba41796f9f001edf9c99630496b92c64ca"}
Sep 11 18:15:40.505081 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.505040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dppvt" event={"ID":"8fe6b25b-33ee-4180-93f7-e486184b0bfe","Type":"ContainerStarted","Data":"413e9a179a7ddb081a7bc4f044e504a63b5ac6ce406e4a75bb8e4471d48cda9c"}
Sep 11 18:15:40.506747 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.506717    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8f6ac10-f51e-453c-8d72-09cfa833f696" containerID="72c84cb82bea311b58f810acc7ea6ae33562b401f2c50395bfa59bacb7d678e8" exitCode=0
Sep 11 18:15:40.506850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.506803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lrgh" event={"ID":"c8f6ac10-f51e-453c-8d72-09cfa833f696","Type":"ContainerDied","Data":"72c84cb82bea311b58f810acc7ea6ae33562b401f2c50395bfa59bacb7d678e8"}
Sep 11 18:15:40.508726 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.508614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cqq8n" event={"ID":"355ecf34-6283-43ff-9bbc-1313be1dea3a","Type":"ContainerStarted","Data":"38d0e850adcba2fa9bd73833d9260f19ad8fef39d244e2b564da86c84b485bae"}
Sep 11 18:15:40.510695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.510667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h" event={"ID":"d81ade46-ec28-4514-8654-19df644ee51b","Type":"ContainerStarted","Data":"97943733a2d5e7c4402543a23d35f6821e1c1a30c82afba45dcedcaa4c91b47f"}
Sep 11 18:15:40.512179 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.512141    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nrdkw" podStartSLOduration=3.748892431 podStartE2EDuration="21.512128507s" totalImagesPullingTime="17.763236076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:40.511848277 +0000 UTC m=+21.608441909" watchObservedRunningTime="2026-09-11 18:15:40.512128507 +0000 UTC m=+21.608722138"
Sep 11 18:15:40.565803 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.565763    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pdmwk" podStartSLOduration=3.775793213 podStartE2EDuration="21.565745301s" totalImagesPullingTime="17.789952088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:40.548650774 +0000 UTC m=+21.645244404" watchObservedRunningTime="2026-09-11 18:15:40.565745301 +0000 UTC m=+21.662338931"
Sep 11 18:15:40.565933 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.565916    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dppvt" podStartSLOduration=3.690493034 podStartE2EDuration="21.565911864s" totalImagesPullingTime="17.87541883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:40.565633314 +0000 UTC m=+21.662226967" watchObservedRunningTime="2026-09-11 18:15:40.565911864 +0000 UTC m=+21.662505495"
Sep 11 18:15:40.601076 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.601017    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9zqsb" podStartSLOduration=7.92413187 podStartE2EDuration="21.600996596s" totalImagesPullingTime="13.676864726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:40.60085302 +0000 UTC m=+21.697446651" watchObservedRunningTime="2026-09-11 18:15:40.600996596 +0000 UTC m=+21.697590227"
Sep 11 18:15:40.601438 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:40.601409    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cqq8n" podStartSLOduration=3.841601687 podStartE2EDuration="21.601401274s" totalImagesPullingTime="17.759799587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:40.58522375 +0000 UTC m=+21.681817380" watchObservedRunningTime="2026-09-11 18:15:40.601401274 +0000 UTC m=+21.697994905"
Sep 11 18:15:41.389559 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:41.389526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:41.389744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:41.389568    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:41.389744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:41.389526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:41.389744 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:41.389679    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:41.389916 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:41.389843    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:41.389972 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:41.389931    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:41.516318 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:41.516230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p" event={"ID":"a2959276-2bdb-41ee-8c15-779298446e02","Type":"ContainerStarted","Data":"78babfc0a2d6b8f39fcac8ae3fa23a0c4eb528b6c3a6121cbc27f22500f7e44d"}
Sep 11 18:15:41.516318 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:41.516297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p" event={"ID":"a2959276-2bdb-41ee-8c15-779298446e02","Type":"ContainerStarted","Data":"42b1112dab36cc0ef70ec4b82c51dfe4deba854de92c35989bf550c9d35fba96"}
Sep 11 18:15:41.517084 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:41.516621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p" event={"ID":"a2959276-2bdb-41ee-8c15-779298446e02","Type":"ContainerStarted","Data":"fd7137944553f648d985eefd591234ddfa4624b40cd1dfed12f3810b96afd625"}
Sep 11 18:15:41.517084 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:41.516717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p" event={"ID":"a2959276-2bdb-41ee-8c15-779298446e02","Type":"ContainerStarted","Data":"3edfc70333aef9bb20d2d3b2ef1bad57094d887e454eefc78071a0f0afe7d600"}
Sep 11 18:15:41.790695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:41.790662    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 18:15:42.357520 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:42.356770    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T18:15:41.790686643Z","UUID":"c85a42d9-8741-4d20-b985-39b54c39e461","Handler":null,"Name":"","Endpoint":""}
Sep 11 18:15:42.358900 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:42.358872    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 18:15:42.359046 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:42.358911    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 18:15:42.520630 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:42.520184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jldjw" event={"ID":"f0a70781-97db-4f2d-9d11-d13d6cfdd56d","Type":"ContainerStarted","Data":"aa2cb98f34fd191556cb6cd58c4bac37d477e6d90717697d06194b7cd8379127"}
Sep 11 18:15:42.522380 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:42.522345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h" event={"ID":"d81ade46-ec28-4514-8654-19df644ee51b","Type":"ContainerStarted","Data":"658e0e35f81dc9d9ea6c8a6d5f8757cf20197ac6eb4314670d2c5b9b608d65a4"}
Sep 11 18:15:42.535507 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:42.535245    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jldjw" podStartSLOduration=5.809047672 podStartE2EDuration="23.535223935s" totalImagesPullingTime="17.726176263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:42.535016808 +0000 UTC m=+23.631610440" watchObservedRunningTime="2026-09-11 18:15:42.535223935 +0000 UTC m=+23.631817562"
Sep 11 18:15:43.389329 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:43.389293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:43.389532 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:43.389408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:43.389532 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:43.389426    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:43.389532 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:43.389445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:43.389532 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:43.389526    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:43.389752 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:43.389646    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:43.493533 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:43.493494    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:43.494167 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:43.494140    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:43.527648 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:43.527611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h" event={"ID":"d81ade46-ec28-4514-8654-19df644ee51b","Type":"ContainerStarted","Data":"a7d66d8d9e3714e4630cd71174733b0bdc396ebe0c476b0e24a0b743950a7b98"}
Sep 11 18:15:43.532377 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:43.532334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p" event={"ID":"a2959276-2bdb-41ee-8c15-779298446e02","Type":"ContainerStarted","Data":"635ddbbb0b5450d488a54664451787044a7971cfff2881dd2b8f4b576965c7df"}
Sep 11 18:15:43.532606 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:43.532577    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:43.533151 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:43.533133    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9zqsb"
Sep 11 18:15:43.562751 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:43.562694    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf67h" podStartSLOduration=4.56661885 podStartE2EDuration="24.562667136s" totalImagesPullingTime="19.996048286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:43.54772738 +0000 UTC m=+24.644321011" watchObservedRunningTime="2026-09-11 18:15:43.562667136 +0000 UTC m=+24.659260768"
Sep 11 18:15:45.389584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:45.389393    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:45.390009 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:45.389393    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:45.390009 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:45.389665    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:45.390009 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:45.389404    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:45.390009 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:45.389737    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:45.390009 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:45.389836    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:45.538633 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:45.538591    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8f6ac10-f51e-453c-8d72-09cfa833f696" containerID="bf6e0e36f46c586c2b2fe119c3c8262c22c61b5c1d28d88bb5e42adbf180be69" exitCode=0
Sep 11 18:15:45.538792 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:45.538684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lrgh" event={"ID":"c8f6ac10-f51e-453c-8d72-09cfa833f696","Type":"ContainerDied","Data":"bf6e0e36f46c586c2b2fe119c3c8262c22c61b5c1d28d88bb5e42adbf180be69"}
Sep 11 18:15:45.542739 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:45.542714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p" event={"ID":"a2959276-2bdb-41ee-8c15-779298446e02","Type":"ContainerStarted","Data":"7f99ebc9aaf9841a2bcbb177e22c9d3db4ee7846419ceac9e9356856a9ebdb03"}
Sep 11 18:15:45.542944 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:45.542926    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:45.543006 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:45.543000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:45.559893 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:45.559868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:45.587600 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:45.587551    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p" podStartSLOduration=8.809360389 podStartE2EDuration="26.587533444s" totalImagesPullingTime="17.778173055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:45.587137284 +0000 UTC m=+26.683730915" watchObservedRunningTime="2026-09-11 18:15:45.587533444 +0000 UTC m=+26.684127075"
Sep 11 18:15:46.545267 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:46.545229    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:46.560789 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:46.560766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:15:47.260251 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:47.260017    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8s7n8"]
Sep 11 18:15:47.260420 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:47.260365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:47.260486 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:47.260467    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:47.273490 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:47.273459    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-plhmr"]
Sep 11 18:15:47.273620 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:47.273582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:47.273696 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:47.273669    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:47.274043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:47.274006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jhdmg"]
Sep 11 18:15:47.274174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:47.274103    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:47.274247 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:47.274194    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:47.549652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:47.549575    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8f6ac10-f51e-453c-8d72-09cfa833f696" containerID="ad44448ad0407585a374fcd649443fb831589ea03467ef60be844be8da777cc5" exitCode=0
Sep 11 18:15:47.550009 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:47.549657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lrgh" event={"ID":"c8f6ac10-f51e-453c-8d72-09cfa833f696","Type":"ContainerDied","Data":"ad44448ad0407585a374fcd649443fb831589ea03467ef60be844be8da777cc5"}
Sep 11 18:15:49.390759 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:49.390721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:49.391580 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:49.390811    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:49.391580 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:49.390905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:49.391580 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:49.391009    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:49.391580 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:49.391068    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:49.391580 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:49.391158    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:49.557553 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:49.557519    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8f6ac10-f51e-453c-8d72-09cfa833f696" containerID="6b298a5d8e36a03a87b8c500660362d34e8243d660f50f85d70c56953acf730c" exitCode=0
Sep 11 18:15:49.557718 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:49.557578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lrgh" event={"ID":"c8f6ac10-f51e-453c-8d72-09cfa833f696","Type":"ContainerDied","Data":"6b298a5d8e36a03a87b8c500660362d34e8243d660f50f85d70c56953acf730c"}
Sep 11 18:15:51.389831 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:51.389599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:51.390277 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:51.389599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:51.390277 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:51.389936    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:51.390277 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:51.389635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:51.390277 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:51.390005    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:51.390277 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:51.390122    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:53.039676 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:53.039635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:53.040262 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:53.039719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:53.040262 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.039804    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:53.040262 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.039825    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:53.040262 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.039880    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret podName:1649793f-bd76-49a7-9b4e-61473e8857b2 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:25.03986409 +0000 UTC m=+66.136457699 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret") pod "global-pull-secret-syncer-8s7n8" (UID: "1649793f-bd76-49a7-9b4e-61473e8857b2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:53.040262 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.039901    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs podName:cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:25.039890234 +0000 UTC m=+66.136483844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs") pod "network-metrics-daemon-plhmr" (UID: "cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:53.242044 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:53.242003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7vt\" (UniqueName: \"kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt\") pod \"network-check-target-jhdmg\" (UID: \"c5329603-85e2-4ad4-a14e-4b37e0b1c105\") " pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:53.242242 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.242213    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:53.242242 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.242241    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:53.242361 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.242255    2580 projected.go:196] Error preparing data for projected volume kube-api-access-ww7vt for pod openshift-network-diagnostics/network-check-target-jhdmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:53.242361 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.242318    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt podName:c5329603-85e2-4ad4-a14e-4b37e0b1c105 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:25.242303821 +0000 UTC m=+66.338897448 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww7vt" (UniqueName: "kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt") pod "network-check-target-jhdmg" (UID: "c5329603-85e2-4ad4-a14e-4b37e0b1c105") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:53.390167 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:53.390126    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:53.390167 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:53.390154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:53.390440 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.390285    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8s7n8" podUID="1649793f-bd76-49a7-9b4e-61473e8857b2"
Sep 11 18:15:53.390440 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.390367    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plhmr" podUID="cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53"
Sep 11 18:15:53.390554 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:53.390449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:53.390605 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:53.390562    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhdmg" podUID="c5329603-85e2-4ad4-a14e-4b37e0b1c105"
Sep 11 18:15:53.571708 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:53.571671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lrgh" event={"ID":"c8f6ac10-f51e-453c-8d72-09cfa833f696","Type":"ContainerStarted","Data":"3d1f206f0f000ba61a80ac3116662171966e3a899ba79e737cadad01a6f49b3c"}
Sep 11 18:15:54.200651 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.200620    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-7.ec2.internal" event="NodeReady"
Sep 11 18:15:54.201030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.200745    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 18:15:54.239593 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.239559    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86766dfb79-rqf92"]
Sep 11 18:15:54.241412 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.241395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.243564 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.243539    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 18:15:54.243795 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.243778    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 18:15:54.243853 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.243838    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vxws4\""
Sep 11 18:15:54.243891 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.243847    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 18:15:54.249069 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.249043    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 18:15:54.251091 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.251068    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h87pc"]
Sep 11 18:15:54.255479 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.255454    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rnzmv"]
Sep 11 18:15:54.255589 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.255572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.257220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.257174    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86766dfb79-rqf92"]
Sep 11 18:15:54.257376 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.257281    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rnzmv"
Sep 11 18:15:54.258070 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.257953    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 18:15:54.258070 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.257979    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 18:15:54.258070 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.257964    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9ltz\""
Sep 11 18:15:54.259686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.259655    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kjt2v\""
Sep 11 18:15:54.259686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.259685    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 18:15:54.260005 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.259729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 18:15:54.260005 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.259686    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 18:15:54.265398 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.265371    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h87pc"]
Sep 11 18:15:54.267974 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.267948    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rnzmv"]
Sep 11 18:15:54.350420 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/030d0519-adf2-40e3-a456-7d80868f9725-config-volume\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.350420 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ff90c73-d339-499d-95a6-b2aa2657f85d-ca-trust-extracted\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.350642 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nmt\" (UniqueName: \"kubernetes.io/projected/15a582f5-b3ab-4ba3-aed1-6f86d5308a2f-kube-api-access-r7nmt\") pod \"ingress-canary-rnzmv\" (UID: \"15a582f5-b3ab-4ba3-aed1-6f86d5308a2f\") " pod="openshift-ingress-canary/ingress-canary-rnzmv"
Sep 11 18:15:54.350642 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-trusted-ca\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.350642 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15a582f5-b3ab-4ba3-aed1-6f86d5308a2f-cert\") pod \"ingress-canary-rnzmv\" (UID: \"15a582f5-b3ab-4ba3-aed1-6f86d5308a2f\") " pod="openshift-ingress-canary/ingress-canary-rnzmv"
Sep 11 18:15:54.350642 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/030d0519-adf2-40e3-a456-7d80868f9725-tmp-dir\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.350642 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.350796 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrxlj\" (UniqueName: \"kubernetes.io/projected/030d0519-adf2-40e3-a456-7d80868f9725-kube-api-access-xrxlj\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.350796 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-certificates\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.350796 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-image-registry-private-configuration\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.350796 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.350916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-installation-pull-secrets\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.350916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znbrq\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-kube-api-access-znbrq\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.350916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.350845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-bound-sa-token\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.451785 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.451703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-image-registry-private-configuration\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.451785 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.451740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.451785 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.451764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-installation-pull-secrets\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.451787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znbrq\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-kube-api-access-znbrq\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.451820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-bound-sa-token\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.451847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/030d0519-adf2-40e3-a456-7d80868f9725-config-volume\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.451871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ff90c73-d339-499d-95a6-b2aa2657f85d-ca-trust-extracted\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:54.451883    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.451899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nmt\" (UniqueName: \"kubernetes.io/projected/15a582f5-b3ab-4ba3-aed1-6f86d5308a2f-kube-api-access-r7nmt\") pod \"ingress-canary-rnzmv\" (UID: \"15a582f5-b3ab-4ba3-aed1-6f86d5308a2f\") " pod="openshift-ingress-canary/ingress-canary-rnzmv"
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:54.451910    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86766dfb79-rqf92: secret "image-registry-tls" not found
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.451924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-trusted-ca\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:54.451966    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls podName:6ff90c73-d339-499d-95a6-b2aa2657f85d nodeName:}" failed. No retries permitted until 2026-09-11 18:15:54.951946667 +0000 UTC m=+36.048540289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls") pod "image-registry-86766dfb79-rqf92" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d") : secret "image-registry-tls" not found
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.451986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15a582f5-b3ab-4ba3-aed1-6f86d5308a2f-cert\") pod \"ingress-canary-rnzmv\" (UID: \"15a582f5-b3ab-4ba3-aed1-6f86d5308a2f\") " pod="openshift-ingress-canary/ingress-canary-rnzmv"
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.452015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/030d0519-adf2-40e3-a456-7d80868f9725-tmp-dir\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.452048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.452053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.452673 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.452076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrxlj\" (UniqueName: \"kubernetes.io/projected/030d0519-adf2-40e3-a456-7d80868f9725-kube-api-access-xrxlj\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.452673 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.452131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-certificates\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.452673 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:54.452330    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:15:54.452673 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:54.452398    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls podName:030d0519-adf2-40e3-a456-7d80868f9725 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:54.952380316 +0000 UTC m=+36.048973935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls") pod "dns-default-h87pc" (UID: "030d0519-adf2-40e3-a456-7d80868f9725") : secret "dns-default-metrics-tls" not found
Sep 11 18:15:54.452673 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.452447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/030d0519-adf2-40e3-a456-7d80868f9725-tmp-dir\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.452673 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.452486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/030d0519-adf2-40e3-a456-7d80868f9725-config-volume\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.452943 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.452730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-certificates\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.452943 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.452757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ff90c73-d339-499d-95a6-b2aa2657f85d-ca-trust-extracted\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.453063 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.453043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-trusted-ca\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.456423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.456390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15a582f5-b3ab-4ba3-aed1-6f86d5308a2f-cert\") pod \"ingress-canary-rnzmv\" (UID: \"15a582f5-b3ab-4ba3-aed1-6f86d5308a2f\") " pod="openshift-ingress-canary/ingress-canary-rnzmv"
Sep 11 18:15:54.456423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.456406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-installation-pull-secrets\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.456576 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.456397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-image-registry-private-configuration\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.464534 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.464497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-bound-sa-token\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.464666 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.464642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrxlj\" (UniqueName: \"kubernetes.io/projected/030d0519-adf2-40e3-a456-7d80868f9725-kube-api-access-xrxlj\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.464737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.464722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znbrq\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-kube-api-access-znbrq\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.464773 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.464742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nmt\" (UniqueName: \"kubernetes.io/projected/15a582f5-b3ab-4ba3-aed1-6f86d5308a2f-kube-api-access-r7nmt\") pod \"ingress-canary-rnzmv\" (UID: \"15a582f5-b3ab-4ba3-aed1-6f86d5308a2f\") " pod="openshift-ingress-canary/ingress-canary-rnzmv"
Sep 11 18:15:54.572299 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.572268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rnzmv"
Sep 11 18:15:54.576941 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.576908    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8f6ac10-f51e-453c-8d72-09cfa833f696" containerID="3d1f206f0f000ba61a80ac3116662171966e3a899ba79e737cadad01a6f49b3c" exitCode=0
Sep 11 18:15:54.577082 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.576956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lrgh" event={"ID":"c8f6ac10-f51e-453c-8d72-09cfa833f696","Type":"ContainerDied","Data":"3d1f206f0f000ba61a80ac3116662171966e3a899ba79e737cadad01a6f49b3c"}
Sep 11 18:15:54.862033 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.861808    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rnzmv"]
Sep 11 18:15:54.864815 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:15:54.864782    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a582f5_b3ab_4ba3_aed1_6f86d5308a2f.slice/crio-f5a4496911e11d81d058116526846820477af11fae8916cd18638d31b55a2f34 WatchSource:0}: Error finding container f5a4496911e11d81d058116526846820477af11fae8916cd18638d31b55a2f34: Status 404 returned error can't find the container with id f5a4496911e11d81d058116526846820477af11fae8916cd18638d31b55a2f34
Sep 11 18:15:54.954977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.954941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:54.955146 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:54.955003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:54.955146 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:54.955088    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:15:54.955250 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:54.955152    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls podName:030d0519-adf2-40e3-a456-7d80868f9725 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:55.955134166 +0000 UTC m=+37.051727775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls") pod "dns-default-h87pc" (UID: "030d0519-adf2-40e3-a456-7d80868f9725") : secret "dns-default-metrics-tls" not found
Sep 11 18:15:54.955250 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:54.955092    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:15:54.955250 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:54.955190    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86766dfb79-rqf92: secret "image-registry-tls" not found
Sep 11 18:15:54.955343 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:54.955253    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls podName:6ff90c73-d339-499d-95a6-b2aa2657f85d nodeName:}" failed. No retries permitted until 2026-09-11 18:15:55.95523631 +0000 UTC m=+37.051829926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls") pod "image-registry-86766dfb79-rqf92" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d") : secret "image-registry-tls" not found
Sep 11 18:15:55.390129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.390087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:15:55.390599 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.390087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:15:55.390599 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.390096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:15:55.393194 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.393172    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:15:55.393350 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.393219    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:15:55.393350 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.393263    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-78r7v\""
Sep 11 18:15:55.393350 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.393336    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:15:55.393585 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.393569    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28rql\""
Sep 11 18:15:55.393660 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.393645    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:15:55.581483 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.581411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rnzmv" event={"ID":"15a582f5-b3ab-4ba3-aed1-6f86d5308a2f","Type":"ContainerStarted","Data":"f5a4496911e11d81d058116526846820477af11fae8916cd18638d31b55a2f34"}
Sep 11 18:15:55.585134 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.585089    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8f6ac10-f51e-453c-8d72-09cfa833f696" containerID="239b17b600430e913095d256388e4f01f08ec52762738ddbdac8bcaa49a0eb84" exitCode=0
Sep 11 18:15:55.585134 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.585139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lrgh" event={"ID":"c8f6ac10-f51e-453c-8d72-09cfa833f696","Type":"ContainerDied","Data":"239b17b600430e913095d256388e4f01f08ec52762738ddbdac8bcaa49a0eb84"}
Sep 11 18:15:55.962490 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.962394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:55.962634 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:55.962493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:55.962634 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:55.962572    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:15:55.962634 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:55.962605    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86766dfb79-rqf92: secret "image-registry-tls" not found
Sep 11 18:15:55.962763 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:55.962577    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:15:55.962763 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:55.962674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls podName:6ff90c73-d339-499d-95a6-b2aa2657f85d nodeName:}" failed. No retries permitted until 2026-09-11 18:15:57.96265215 +0000 UTC m=+39.059245768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls") pod "image-registry-86766dfb79-rqf92" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d") : secret "image-registry-tls" not found
Sep 11 18:15:55.962763 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:55.962724    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls podName:030d0519-adf2-40e3-a456-7d80868f9725 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:57.962703746 +0000 UTC m=+39.059297361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls") pod "dns-default-h87pc" (UID: "030d0519-adf2-40e3-a456-7d80868f9725") : secret "dns-default-metrics-tls" not found
Sep 11 18:15:56.591520 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:56.591477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lrgh" event={"ID":"c8f6ac10-f51e-453c-8d72-09cfa833f696","Type":"ContainerStarted","Data":"5c8a32a85a5d11272dbcf32284b2c948c73cf69e8084c8b4f0ee5cc6f87b4257"}
Sep 11 18:15:56.616515 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:56.616461    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2lrgh" podStartSLOduration=9.369773273 podStartE2EDuration="37.616443272s" totalImagesPullingTime="28.246669999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:56.615563881 +0000 UTC m=+37.712157511" watchObservedRunningTime="2026-09-11 18:15:56.616443272 +0000 UTC m=+37.713036905"
Sep 11 18:15:56.771769 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:56.771733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:15:57.977450 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:57.977405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:15:57.977908 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:57.977476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:15:57.977908 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:57.977554    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:15:57.977908 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:57.977573    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:15:57.977908 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:57.977585    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86766dfb79-rqf92: secret "image-registry-tls" not found
Sep 11 18:15:57.977908 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:57.977617    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls podName:030d0519-adf2-40e3-a456-7d80868f9725 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:01.977599053 +0000 UTC m=+43.074192661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls") pod "dns-default-h87pc" (UID: "030d0519-adf2-40e3-a456-7d80868f9725") : secret "dns-default-metrics-tls" not found
Sep 11 18:15:57.977908 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:15:57.977631    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls podName:6ff90c73-d339-499d-95a6-b2aa2657f85d nodeName:}" failed. No retries permitted until 2026-09-11 18:16:01.97762493 +0000 UTC m=+43.074218538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls") pod "image-registry-86766dfb79-rqf92" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d") : secret "image-registry-tls" not found
Sep 11 18:15:58.510786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:58.510744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:15:58.598503 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:58.598466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rnzmv" event={"ID":"15a582f5-b3ab-4ba3-aed1-6f86d5308a2f","Type":"ContainerStarted","Data":"47208c4c1d047179abadaa12e0466118f4e3accd45d1a01f5c79504f56b8a6ee"}
Sep 11 18:15:58.615187 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:15:58.615126    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rnzmv" podStartSLOduration=1.436402294 podStartE2EDuration="4.615108382s" totalImagesPullingTime="3.178706088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:58.614827592 +0000 UTC m=+39.711421223" watchObservedRunningTime="2026-09-11 18:15:58.615108382 +0000 UTC m=+39.711702013"
Sep 11 18:16:01.187973 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:01.187940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:01.247426 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:01.247376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:01.319749 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:01.319721    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:02.008237 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:02.008177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:16:02.008415 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:02.008255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:16:02.008415 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:02.008346    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:02.008415 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:02.008374    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86766dfb79-rqf92: secret "image-registry-tls" not found
Sep 11 18:16:02.008415 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:02.008353    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:02.008543 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:02.008424    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls podName:030d0519-adf2-40e3-a456-7d80868f9725 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:10.008411214 +0000 UTC m=+51.105004823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls") pod "dns-default-h87pc" (UID: "030d0519-adf2-40e3-a456-7d80868f9725") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:02.008543 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:02.008437    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls podName:6ff90c73-d339-499d-95a6-b2aa2657f85d nodeName:}" failed. No retries permitted until 2026-09-11 18:16:10.008431196 +0000 UTC m=+51.105024805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls") pod "image-registry-86766dfb79-rqf92" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d") : secret "image-registry-tls" not found
Sep 11 18:16:02.117239 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:02.117187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:02.173993 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:02.173957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:03.705822 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:03.705783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:03.768649 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:03.768615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:08.416574 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:08.416535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:08.489636 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:08.489596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:10.062630 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:10.062577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:16:10.063251 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:10.062757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:16:10.063251 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:10.062761    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:10.063251 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:10.062815    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86766dfb79-rqf92: secret "image-registry-tls" not found
Sep 11 18:16:10.063251 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:10.062825    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:10.063251 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:10.062877    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls podName:6ff90c73-d339-499d-95a6-b2aa2657f85d nodeName:}" failed. No retries permitted until 2026-09-11 18:16:26.062863828 +0000 UTC m=+67.159457437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls") pod "image-registry-86766dfb79-rqf92" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d") : secret "image-registry-tls" not found
Sep 11 18:16:10.063251 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:10.062890    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls podName:030d0519-adf2-40e3-a456-7d80868f9725 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:26.062885077 +0000 UTC m=+67.159478686 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls") pod "dns-default-h87pc" (UID: "030d0519-adf2-40e3-a456-7d80868f9725") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:18.567577 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:18.567542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrh4p"
Sep 11 18:16:18.721869 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:18.721837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:18.822240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:18.822115    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:21.475284 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:21.475238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:21.575658 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:21.575620    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:25.068295 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.068255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:16:25.068787 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.068327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:16:25.070589 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.070565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:16:25.070710 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.070635    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:16:25.079506 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:25.079476    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 18:16:25.079601 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:25.079547    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs podName:cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:29.079529594 +0000 UTC m=+130.176123209 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs") pod "network-metrics-daemon-plhmr" (UID: "cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53") : secret "metrics-daemon-secret" not found
Sep 11 18:16:25.082579 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.082549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1649793f-bd76-49a7-9b4e-61473e8857b2-original-pull-secret\") pod \"global-pull-secret-syncer-8s7n8\" (UID: \"1649793f-bd76-49a7-9b4e-61473e8857b2\") " pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:16:25.100679 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.100646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8s7n8"
Sep 11 18:16:25.148391 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.148351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:25.243865 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.243836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:25.270137 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.270105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7vt\" (UniqueName: \"kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt\") pod \"network-check-target-jhdmg\" (UID: \"c5329603-85e2-4ad4-a14e-4b37e0b1c105\") " pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:16:25.272984 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.272964    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:16:25.283157 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.283137    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:16:25.293499 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.293473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7vt\" (UniqueName: \"kubernetes.io/projected/c5329603-85e2-4ad4-a14e-4b37e0b1c105-kube-api-access-ww7vt\") pod \"network-check-target-jhdmg\" (UID: \"c5329603-85e2-4ad4-a14e-4b37e0b1c105\") " pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:16:25.401976 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.401944    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8s7n8"]
Sep 11 18:16:25.404877 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:16:25.404844    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1649793f_bd76_49a7_9b4e_61473e8857b2.slice/crio-758f1ac0761a7191c5fab935b15168e894f6f28bdaab228dfd6b690685f7de0d WatchSource:0}: Error finding container 758f1ac0761a7191c5fab935b15168e894f6f28bdaab228dfd6b690685f7de0d: Status 404 returned error can't find the container with id 758f1ac0761a7191c5fab935b15168e894f6f28bdaab228dfd6b690685f7de0d
Sep 11 18:16:25.408850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.408829    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-78r7v\""
Sep 11 18:16:25.416889 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.416863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:16:25.664109 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.664024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8s7n8" event={"ID":"1649793f-bd76-49a7-9b4e-61473e8857b2","Type":"ContainerStarted","Data":"758f1ac0761a7191c5fab935b15168e894f6f28bdaab228dfd6b690685f7de0d"}
Sep 11 18:16:25.718270 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:25.718222    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jhdmg"]
Sep 11 18:16:25.721492 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:16:25.721464    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5329603_85e2_4ad4_a14e_4b37e0b1c105.slice/crio-cbd83430511979feb63d89ddeaf4f3045850dffdf1c48e3590d4ea228ecddb41 WatchSource:0}: Error finding container cbd83430511979feb63d89ddeaf4f3045850dffdf1c48e3590d4ea228ecddb41: Status 404 returned error can't find the container with id cbd83430511979feb63d89ddeaf4f3045850dffdf1c48e3590d4ea228ecddb41
Sep 11 18:16:26.074732 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:26.074644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:16:26.074732 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:26.074720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:16:26.075194 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:26.074820    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:26.075194 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:26.074847    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86766dfb79-rqf92: secret "image-registry-tls" not found
Sep 11 18:16:26.075194 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:26.074901    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls podName:6ff90c73-d339-499d-95a6-b2aa2657f85d nodeName:}" failed. No retries permitted until 2026-09-11 18:16:58.074886308 +0000 UTC m=+99.171479917 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls") pod "image-registry-86766dfb79-rqf92" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d") : secret "image-registry-tls" not found
Sep 11 18:16:26.075194 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:26.074827    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:26.075194 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:26.074982    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls podName:030d0519-adf2-40e3-a456-7d80868f9725 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:58.074966856 +0000 UTC m=+99.171560467 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls") pod "dns-default-h87pc" (UID: "030d0519-adf2-40e3-a456-7d80868f9725") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:26.668287 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:26.668242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jhdmg" event={"ID":"c5329603-85e2-4ad4-a14e-4b37e0b1c105","Type":"ContainerStarted","Data":"cbd83430511979feb63d89ddeaf4f3045850dffdf1c48e3590d4ea228ecddb41"}
Sep 11 18:16:29.688108 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:29.688071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:29.784566 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:29.784510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:30.679605 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:30.679569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8s7n8" event={"ID":"1649793f-bd76-49a7-9b4e-61473e8857b2","Type":"ContainerStarted","Data":"b042dd9582276a3ccca2cee581bc0b7b54f199f261d9ca76f87d7f4fb17adea1"}
Sep 11 18:16:30.681190 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:30.681164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jhdmg" event={"ID":"c5329603-85e2-4ad4-a14e-4b37e0b1c105","Type":"ContainerStarted","Data":"2dd90b1bfedc2f5957ef2d2238b236b10cfe8564a2da2ac7d0caa6ba02788e25"}
Sep 11 18:16:30.681363 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:30.681327    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:16:30.696180 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:30.696126    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8s7n8" podStartSLOduration=67.058716496 podStartE2EDuration="1m11.696110538s" totalImagesPullingTime="4.637394042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:30.695490584 +0000 UTC m=+71.792084215" watchObservedRunningTime="2026-09-11 18:16:30.696110538 +0000 UTC m=+71.792704169"
Sep 11 18:16:34.456257 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:34.456218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:34.855354 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:34.855268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:35.226298 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.226239    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jhdmg" podStartSLOduration=71.910605636 podStartE2EDuration="1m16.226193719s" totalImagesPullingTime="4.315588083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:30.709616728 +0000 UTC m=+71.806210358" watchObservedRunningTime="2026-09-11 18:16:35.226193719 +0000 UTC m=+76.322787349"
Sep 11 18:16:35.226481 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.226355    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"]
Sep 11 18:16:35.230405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.230384    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:35.240974 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.240951    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:16:35.242059 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.242044    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 18:16:35.243359 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.243344    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jjzt8\""
Sep 11 18:16:35.243510 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.243501    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 18:16:35.281921 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.281884    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"]
Sep 11 18:16:35.345515 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.345472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g979v\" (UniqueName: \"kubernetes.io/projected/ffa43ebb-93a0-40eb-80e6-ac32902293d8-kube-api-access-g979v\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:35.345710 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.345545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:35.446536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.446491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:35.446734 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.446634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g979v\" (UniqueName: \"kubernetes.io/projected/ffa43ebb-93a0-40eb-80e6-ac32902293d8-kube-api-access-g979v\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:35.446734 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:35.446648    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:16:35.446734 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:35.446718    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls podName:ffa43ebb-93a0-40eb-80e6-ac32902293d8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:35.946702322 +0000 UTC m=+77.043295954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nvff4" (UID: "ffa43ebb-93a0-40eb-80e6-ac32902293d8") : secret "samples-operator-tls" not found
Sep 11 18:16:35.462850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.462816    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g979v\" (UniqueName: \"kubernetes.io/projected/ffa43ebb-93a0-40eb-80e6-ac32902293d8-kube-api-access-g979v\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:35.480960 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.480855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:35.537516 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.537477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:35.951090 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:35.951049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:35.951291 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:35.951229    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:16:35.951335 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:35.951297    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls podName:ffa43ebb-93a0-40eb-80e6-ac32902293d8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:36.951280706 +0000 UTC m=+78.047874320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nvff4" (UID: "ffa43ebb-93a0-40eb-80e6-ac32902293d8") : secret "samples-operator-tls" not found
Sep 11 18:16:36.958822 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:36.958788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:36.959280 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:36.958947    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:16:36.959280 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:36.959021    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls podName:ffa43ebb-93a0-40eb-80e6-ac32902293d8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:38.959006218 +0000 UTC m=+80.055599832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nvff4" (UID: "ffa43ebb-93a0-40eb-80e6-ac32902293d8") : secret "samples-operator-tls" not found
Sep 11 18:16:37.536579 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:37.536541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:37.591162 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:37.591126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:38.975503 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:38.975459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:38.975905 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:38.975616    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:16:38.975905 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:38.975696    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls podName:ffa43ebb-93a0-40eb-80e6-ac32902293d8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.975680865 +0000 UTC m=+84.072274473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nvff4" (UID: "ffa43ebb-93a0-40eb-80e6-ac32902293d8") : secret "samples-operator-tls" not found
Sep 11 18:16:40.980236 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:40.980176    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nr8s4"]
Sep 11 18:16:40.984492 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:40.984469    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:40.986490 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:40.986468    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 18:16:40.986631 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:40.986619    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 18:16:40.986914 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:40.986895    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 18:16:40.986969 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:40.986901    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 18:16:40.987291 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:40.987275    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fpnpf\""
Sep 11 18:16:40.994547 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:40.994519    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nr8s4"]
Sep 11 18:16:40.994796 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:40.994769    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:41.058596 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.058561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:41.088567 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.088536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgmv\" (UniqueName: \"kubernetes.io/projected/5c53d821-2989-42fb-8d84-07d413795f9c-kube-api-access-9tgmv\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.088666 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.088575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c53d821-2989-42fb-8d84-07d413795f9c-crio-socket\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.088709 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.088688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c53d821-2989-42fb-8d84-07d413795f9c-data-volume\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.088752 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.088726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.088809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.088776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.189807 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.189769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c53d821-2989-42fb-8d84-07d413795f9c-crio-socket\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.189984 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.189857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c53d821-2989-42fb-8d84-07d413795f9c-data-volume\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.189984 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.189883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.189984 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.189908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c53d821-2989-42fb-8d84-07d413795f9c-crio-socket\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.189984 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.189902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.190183 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:41.189990    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:16:41.190183 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.189996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgmv\" (UniqueName: \"kubernetes.io/projected/5c53d821-2989-42fb-8d84-07d413795f9c-kube-api-access-9tgmv\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.190183 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:41.190012    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:41.190183 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:41.190066    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm podName:5c53d821-2989-42fb-8d84-07d413795f9c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:41.690044836 +0000 UTC m=+82.786638475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nr8s4" (UID: "5c53d821-2989-42fb-8d84-07d413795f9c") : configmap "kube-rbac-proxy" not found
Sep 11 18:16:41.190183 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:41.190143    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls podName:5c53d821-2989-42fb-8d84-07d413795f9c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:41.690125328 +0000 UTC m=+82.786718938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nr8s4" (UID: "5c53d821-2989-42fb-8d84-07d413795f9c") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:41.190376 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.190237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c53d821-2989-42fb-8d84-07d413795f9c-data-volume\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.205522 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.205497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgmv\" (UniqueName: \"kubernetes.io/projected/5c53d821-2989-42fb-8d84-07d413795f9c-kube-api-access-9tgmv\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.263467 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.263321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:41.263467 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.263440    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r"]
Sep 11 18:16:41.266463 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.266449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r"
Sep 11 18:16:41.268573 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.268558    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 18:16:41.268758 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.268745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dtvp9\""
Sep 11 18:16:41.268803 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.268760    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 18:16:41.278448 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.278423    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r"]
Sep 11 18:16:41.357792 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.357754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:41.391245 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.391186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k29qx\" (UniqueName: \"kubernetes.io/projected/0d74c37f-f474-4ad1-b428-f4ebdea92260-kube-api-access-k29qx\") pod \"migrator-8b4d697c5-xd46r\" (UID: \"0d74c37f-f474-4ad1-b428-f4ebdea92260\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r"
Sep 11 18:16:41.458700 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.458656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:41.491632 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.491593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k29qx\" (UniqueName: \"kubernetes.io/projected/0d74c37f-f474-4ad1-b428-f4ebdea92260-kube-api-access-k29qx\") pod \"migrator-8b4d697c5-xd46r\" (UID: \"0d74c37f-f474-4ad1-b428-f4ebdea92260\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r"
Sep 11 18:16:41.500809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.500778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k29qx\" (UniqueName: \"kubernetes.io/projected/0d74c37f-f474-4ad1-b428-f4ebdea92260-kube-api-access-k29qx\") pod \"migrator-8b4d697c5-xd46r\" (UID: \"0d74c37f-f474-4ad1-b428-f4ebdea92260\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r"
Sep 11 18:16:41.575044 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.574958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r"
Sep 11 18:16:41.694293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.694241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.694293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.694290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:41.694518 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:41.694366    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:16:41.694518 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:41.694394    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:41.694518 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:41.694422    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm podName:5c53d821-2989-42fb-8d84-07d413795f9c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.694406352 +0000 UTC m=+83.790999961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nr8s4" (UID: "5c53d821-2989-42fb-8d84-07d413795f9c") : configmap "kube-rbac-proxy" not found
Sep 11 18:16:41.694518 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:41.694453    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls podName:5c53d821-2989-42fb-8d84-07d413795f9c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.694439911 +0000 UTC m=+83.791033520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nr8s4" (UID: "5c53d821-2989-42fb-8d84-07d413795f9c") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:41.721972 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.721940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:41.789028 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.789002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:41.848496 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:41.848409    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r"]
Sep 11 18:16:41.851397 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:16:41.851366    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d74c37f_f474_4ad1_b428_f4ebdea92260.slice/crio-d155131920866066341df87030ad923beb050b223eaad3f27ca9fb524c1f040f WatchSource:0}: Error finding container d155131920866066341df87030ad923beb050b223eaad3f27ca9fb524c1f040f: Status 404 returned error can't find the container with id d155131920866066341df87030ad923beb050b223eaad3f27ca9fb524c1f040f
Sep 11 18:16:42.153342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:42.153303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:42.214713 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:42.214688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:42.434038 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:42.433950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:42.501300 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:42.501264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:42.703217 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:42.703100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:42.703217 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:42.703149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:42.703467 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:42.703275    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:16:42.703467 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:42.703285    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:42.703467 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:42.703343    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm podName:5c53d821-2989-42fb-8d84-07d413795f9c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:44.703324949 +0000 UTC m=+85.799918558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nr8s4" (UID: "5c53d821-2989-42fb-8d84-07d413795f9c") : configmap "kube-rbac-proxy" not found
Sep 11 18:16:42.703467 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:42.703358    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls podName:5c53d821-2989-42fb-8d84-07d413795f9c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:44.703351797 +0000 UTC m=+85.799945405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nr8s4" (UID: "5c53d821-2989-42fb-8d84-07d413795f9c") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:42.710309 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:42.710281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r" event={"ID":"0d74c37f-f474-4ad1-b428-f4ebdea92260","Type":"ContainerStarted","Data":"d155131920866066341df87030ad923beb050b223eaad3f27ca9fb524c1f040f"}
Sep 11 18:16:42.948523 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:42.948492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cqq8n_355ecf34-6283-43ff-9bbc-1313be1dea3a/dns-node-resolver/0.log"
Sep 11 18:16:43.006388 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:43.006355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:43.006559 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:43.006537    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:16:43.006644 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:43.006634    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls podName:ffa43ebb-93a0-40eb-80e6-ac32902293d8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:51.006609316 +0000 UTC m=+92.103202942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nvff4" (UID: "ffa43ebb-93a0-40eb-80e6-ac32902293d8") : secret "samples-operator-tls" not found
Sep 11 18:16:43.714905 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:43.714872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r" event={"ID":"0d74c37f-f474-4ad1-b428-f4ebdea92260","Type":"ContainerStarted","Data":"9820bc40abbfcdcccd475f9e5ca500ae549bb2fff2853734b0439c6ffa8da3c0"}
Sep 11 18:16:43.715284 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:43.714924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r" event={"ID":"0d74c37f-f474-4ad1-b428-f4ebdea92260","Type":"ContainerStarted","Data":"d513387a6f94260a0f20d6dc02e2d27f575eeb8d4f730760563abd1eab56606c"}
Sep 11 18:16:43.730418 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:43.730366    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xd46r" podStartSLOduration=1.619619454 podStartE2EDuration="2.730348738s" totalImagesPullingTime="1.110729284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:43.730147815 +0000 UTC m=+84.826741460" watchObservedRunningTime="2026-09-11 18:16:43.730348738 +0000 UTC m=+84.826942369"
Sep 11 18:16:43.819349 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:43.819319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:43.948820 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:43.948791    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrdkw_adbddaee-40dd-465e-bd6c-7a32d384346a/node-ca/0.log"
Sep 11 18:16:44.073748 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:44.073639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:44.128928 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:44.128899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:44.721836 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:44.721803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:44.721836 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:44.721836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:44.722273 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:44.721924    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:16:44.722273 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:44.721939    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:44.722273 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:44.721979    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm podName:5c53d821-2989-42fb-8d84-07d413795f9c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:48.721962468 +0000 UTC m=+89.818556097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nr8s4" (UID: "5c53d821-2989-42fb-8d84-07d413795f9c") : configmap "kube-rbac-proxy" not found
Sep 11 18:16:44.722273 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:44.721993    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls podName:5c53d821-2989-42fb-8d84-07d413795f9c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:48.72198639 +0000 UTC m=+89.818579999 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nr8s4" (UID: "5c53d821-2989-42fb-8d84-07d413795f9c") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:44.749323 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:44.749293    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rnzmv_15a582f5-b3ab-4ba3-aed1-6f86d5308a2f/serve-healthcheck-canary/0.log"
Sep 11 18:16:45.236535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:45.236496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:46.019760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:46.019727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:46.217876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:46.217843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:46.318988 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:46.318901    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:46.431936 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:46.431901    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:46.528354 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:46.528320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:46.898578 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:46.898543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:47.305705 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:47.305608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:47.698009 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:47.697980    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:47.918195 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:47.918156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:48.312431 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:48.312402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:48.511898 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:48.511851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:48.755269 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:48.755193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:48.755269 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:48.755269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:48.755553 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:48.755373    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:48.755553 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:48.755388    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:16:48.755553 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:48.755451    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm podName:5c53d821-2989-42fb-8d84-07d413795f9c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:56.75542926 +0000 UTC m=+97.852022869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nr8s4" (UID: "5c53d821-2989-42fb-8d84-07d413795f9c") : configmap "kube-rbac-proxy" not found
Sep 11 18:16:48.755553 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:16:48.755467    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls podName:5c53d821-2989-42fb-8d84-07d413795f9c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:56.755461064 +0000 UTC m=+97.852054672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nr8s4" (UID: "5c53d821-2989-42fb-8d84-07d413795f9c") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:48.914494 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:48.914446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:49.499334 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:49.499303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:49.709554 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:49.709516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:49.770576 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:49.770478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:49.910681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:49.910639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:50.310278 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:50.310243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:50.379214 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:50.379172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:50.519564 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:50.519530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:50.907416 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:50.907383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:51.069956 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:51.069916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:51.072366 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:51.072343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa43ebb-93a0-40eb-80e6-ac32902293d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nvff4\" (UID: \"ffa43ebb-93a0-40eb-80e6-ac32902293d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:51.139422 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:51.139372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"
Sep 11 18:16:51.406586 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:51.406543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4"]
Sep 11 18:16:51.506126 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:51.506088    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:51.736238 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:51.736131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4" event={"ID":"ffa43ebb-93a0-40eb-80e6-ac32902293d8","Type":"ContainerStarted","Data":"5c5b6fedecf35927dd44cd17bcaae81860e2db51664cac50143958ffa7c72d9d"}
Sep 11 18:16:51.906327 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:51.906289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:52.086146 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:52.086060    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:52.211763 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:52.211731    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:52.306424 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:52.306387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:52.711949 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:52.711879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:52.906121 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:52.906050    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:53.508655 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:53.508562    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:53.744816 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:53.744778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4" event={"ID":"ffa43ebb-93a0-40eb-80e6-ac32902293d8","Type":"ContainerStarted","Data":"a88fa92a652e026d544bbc0e0109be5a14785b7b20aa2d873ba950465c95ff95"}
Sep 11 18:16:53.745025 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:53.744833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4" event={"ID":"ffa43ebb-93a0-40eb-80e6-ac32902293d8","Type":"ContainerStarted","Data":"03c018625038249e493128a9374f429e36218b33a8d031bb84aad89acc5191be"}
Sep 11 18:16:53.761279 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:53.761157    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nvff4" podStartSLOduration=17.002872767 podStartE2EDuration="18.761139022s" totalImagesPullingTime="1.758266255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:53.760366025 +0000 UTC m=+94.856959656" watchObservedRunningTime="2026-09-11 18:16:53.761139022 +0000 UTC m=+94.857732652"
Sep 11 18:16:53.909471 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:53.909437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:54.104910 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:54.104818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:54.212418 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:54.212386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:54.706671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:54.706625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:55.106517 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:55.106429    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:55.309765 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:55.309731    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:55.706829 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:55.706794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:56.552237 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:56.552187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:56.607085 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:56.607051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:56.710551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:56.710521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:56.812301 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:56.812212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:56.818154 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:56.818125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:56.818232 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:56.818164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:56.818677 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:56.818651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c53d821-2989-42fb-8d84-07d413795f9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:56.820488 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:56.820461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c53d821-2989-42fb-8d84-07d413795f9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr8s4\" (UID: \"5c53d821-2989-42fb-8d84-07d413795f9c\") " pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:56.867847 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:56.867814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:56.893342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:56.893309    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nr8s4"
Sep 11 18:16:57.165162 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:57.165122    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nr8s4"]
Sep 11 18:16:57.167819 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:16:57.167789    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c53d821_2989_42fb_8d84_07d413795f9c.slice/crio-4d8e124b278ad049396b3ffa4d4c6a1acc92fc89b0ffbc0b477f82d8744545f6 WatchSource:0}: Error finding container 4d8e124b278ad049396b3ffa4d4c6a1acc92fc89b0ffbc0b477f82d8744545f6: Status 404 returned error can't find the container with id 4d8e124b278ad049396b3ffa4d4c6a1acc92fc89b0ffbc0b477f82d8744545f6
Sep 11 18:16:57.760599 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:57.760564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nr8s4" event={"ID":"5c53d821-2989-42fb-8d84-07d413795f9c","Type":"ContainerStarted","Data":"f1d38ff4c53f79889e12361e33679cf652e7be2f2dbc64c5c7f681e0307d6ddc"}
Sep 11 18:16:57.760982 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:57.760624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nr8s4" event={"ID":"5c53d821-2989-42fb-8d84-07d413795f9c","Type":"ContainerStarted","Data":"4d8e124b278ad049396b3ffa4d4c6a1acc92fc89b0ffbc0b477f82d8744545f6"}
Sep 11 18:16:58.129321 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.129289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:16:58.129491 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.129366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:16:58.131827 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.131796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/030d0519-adf2-40e3-a456-7d80868f9725-metrics-tls\") pod \"dns-default-h87pc\" (UID: \"030d0519-adf2-40e3-a456-7d80868f9725\") " pod="openshift-dns/dns-default-h87pc"
Sep 11 18:16:58.131950 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.131884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"image-registry-86766dfb79-rqf92\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") " pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:16:58.154329 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.154295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vxws4\""
Sep 11 18:16:58.162277 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.162239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:16:58.168132 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.168111    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9ltz\""
Sep 11 18:16:58.176916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.176847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h87pc"
Sep 11 18:16:58.460317 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.460237    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86766dfb79-rqf92"]
Sep 11 18:16:58.463710 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:16:58.463676    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff90c73_d339_499d_95a6_b2aa2657f85d.slice/crio-34a36c72c92f0819efa87efe8484491cba1c443d8098cae41499d2eae14eede2 WatchSource:0}: Error finding container 34a36c72c92f0819efa87efe8484491cba1c443d8098cae41499d2eae14eede2: Status 404 returned error can't find the container with id 34a36c72c92f0819efa87efe8484491cba1c443d8098cae41499d2eae14eede2
Sep 11 18:16:58.483032 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.482944    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h87pc"]
Sep 11 18:16:58.486078 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:16:58.486043    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030d0519_adf2_40e3_a456_7d80868f9725.slice/crio-da1d1868247834d2987f5109c2f6203aac5b964b5c040f68ec101d353ec899b9 WatchSource:0}: Error finding container da1d1868247834d2987f5109c2f6203aac5b964b5c040f68ec101d353ec899b9: Status 404 returned error can't find the container with id da1d1868247834d2987f5109c2f6203aac5b964b5c040f68ec101d353ec899b9
Sep 11 18:16:58.766134 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.766044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nr8s4" event={"ID":"5c53d821-2989-42fb-8d84-07d413795f9c","Type":"ContainerStarted","Data":"56742f63e1cda2eef4eef934a8c943deb5b8d0a9b831dc838baff07bf0adc520"}
Sep 11 18:16:58.767591 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.767561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h87pc" event={"ID":"030d0519-adf2-40e3-a456-7d80868f9725","Type":"ContainerStarted","Data":"da1d1868247834d2987f5109c2f6203aac5b964b5c040f68ec101d353ec899b9"}
Sep 11 18:16:58.769364 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.769335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86766dfb79-rqf92" event={"ID":"6ff90c73-d339-499d-95a6-b2aa2657f85d","Type":"ContainerStarted","Data":"548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8"}
Sep 11 18:16:58.769489 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.769383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86766dfb79-rqf92" event={"ID":"6ff90c73-d339-499d-95a6-b2aa2657f85d","Type":"ContainerStarted","Data":"34a36c72c92f0819efa87efe8484491cba1c443d8098cae41499d2eae14eede2"}
Sep 11 18:16:58.769489 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.769434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:16:58.790515 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:58.790454    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86766dfb79-rqf92" podStartSLOduration=99.790430582 podStartE2EDuration="1m39.790430582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:58.789380495 +0000 UTC m=+99.885974129" watchObservedRunningTime="2026-09-11 18:16:58.790430582 +0000 UTC m=+99.887024212"
Sep 11 18:16:59.775682 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:59.775578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nr8s4" event={"ID":"5c53d821-2989-42fb-8d84-07d413795f9c","Type":"ContainerStarted","Data":"4314533eefa488069a1aaadfb23ceea571e71a5aaa137e595dcc8894b2078055"}
Sep 11 18:16:59.795348 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:16:59.795291    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nr8s4" podStartSLOduration=17.687067576 podStartE2EDuration="19.795275936s" totalImagesPullingTime="2.10820836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:16:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:59.794054248 +0000 UTC m=+100.890647889" watchObservedRunningTime="2026-09-11 18:16:59.795275936 +0000 UTC m=+100.891869578"
Sep 11 18:17:00.780916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:00.780885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h87pc" event={"ID":"030d0519-adf2-40e3-a456-7d80868f9725","Type":"ContainerStarted","Data":"cf9c0d8b5ebddde3bb0a2c9adea97130692caf849071997a7d60d5e3988d1754"}
Sep 11 18:17:00.781391 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:00.780938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h87pc" event={"ID":"030d0519-adf2-40e3-a456-7d80868f9725","Type":"ContainerStarted","Data":"91890dc35c00a7749caea37c61ec83e835a74f6c3e5a68ec543e4a5e606394df"}
Sep 11 18:17:00.781391 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:00.781236    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h87pc"
Sep 11 18:17:00.797880 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:00.797830    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h87pc" podStartSLOduration=65.115256615 podStartE2EDuration="1m6.797814404s" totalImagesPullingTime="1.682557789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:00.796847832 +0000 UTC m=+101.893441463" watchObservedRunningTime="2026-09-11 18:17:00.797814404 +0000 UTC m=+101.894408035"
Sep 11 18:17:01.420696 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:01.420658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:01.518671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:01.518638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:01.577258 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:01.577224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:01.686493 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:01.686413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jhdmg"
Sep 11 18:17:01.713368 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:01.713339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:01.896653 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:01.896621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:01.997864 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:01.997766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:02.111142 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.111111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:02.320673 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.320592    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:02.588450 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.588328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:02.588910 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.588887    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"]
Sep 11 18:17:02.607474 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.607440    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"]
Sep 11 18:17:02.607658 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.607537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"
Sep 11 18:17:02.611021 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.610994    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-g7p9w\""
Sep 11 18:17:02.611266 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.610994    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 18:17:02.611535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.611520    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 18:17:02.618451 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.618415    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"]
Sep 11 18:17:02.618584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.618473    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"]
Sep 11 18:17:02.618584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.618557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.620775 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.620691    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 18:17:02.620961 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.620945    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 18:17:02.621044 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.620985    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 18:17:02.621103 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.621044    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 18:17:02.621103 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.621047    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 18:17:02.621213 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.621134    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 18:17:02.621213 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.621175    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 18:17:02.693755 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.693722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:02.765838 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.765802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a73a24f-fb75-42c8-96cb-8e8a809c586d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-grkvb\" (UID: \"0a73a24f-fb75-42c8-96cb-8e8a809c586d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"
Sep 11 18:17:02.766023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.765842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.766023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.765890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.766023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.765917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a73a24f-fb75-42c8-96cb-8e8a809c586d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-grkvb\" (UID: \"0a73a24f-fb75-42c8-96cb-8e8a809c586d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"
Sep 11 18:17:02.766023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.765988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-ca\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.766233 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.766034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-hub\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.766233 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.766075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxv7m\" (UniqueName: \"kubernetes.io/projected/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-kube-api-access-wxv7m\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.766233 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.766130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.769045 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.769005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:02.866723 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.866626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-ca\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.866723 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.866670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-hub\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.866945 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.866789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv7m\" (UniqueName: \"kubernetes.io/projected/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-kube-api-access-wxv7m\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.866945 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.866865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.866945 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.866899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a73a24f-fb75-42c8-96cb-8e8a809c586d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-grkvb\" (UID: \"0a73a24f-fb75-42c8-96cb-8e8a809c586d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"
Sep 11 18:17:02.866945 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.866920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.866945 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.866945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.867176 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.866965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a73a24f-fb75-42c8-96cb-8e8a809c586d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-grkvb\" (UID: \"0a73a24f-fb75-42c8-96cb-8e8a809c586d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"
Sep 11 18:17:02.867757 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.867676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.867757 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.867713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a73a24f-fb75-42c8-96cb-8e8a809c586d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-grkvb\" (UID: \"0a73a24f-fb75-42c8-96cb-8e8a809c586d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"
Sep 11 18:17:02.869472 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.869437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-ca\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.869598 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.869487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-hub\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.869780 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.869750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.869872 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.869823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a73a24f-fb75-42c8-96cb-8e8a809c586d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-grkvb\" (UID: \"0a73a24f-fb75-42c8-96cb-8e8a809c586d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"
Sep 11 18:17:02.870073 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.870055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.886073 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.886039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxv7m\" (UniqueName: \"kubernetes.io/projected/ea3b6a2d-2118-4403-96e4-2e9e4f9eec54-kube-api-access-wxv7m\") pod \"cluster-proxy-proxy-agent-f955f556b-dt78v\" (UID: \"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:02.919071 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.919035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"
Sep 11 18:17:02.937924 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:02.937890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"
Sep 11 18:17:03.223585 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:03.223432    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb"]
Sep 11 18:17:03.225958 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:17:03.225925    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a73a24f_fb75_42c8_96cb_8e8a809c586d.slice/crio-7d054730a14ab269983fffe0a33d26143298691554006b7a90ee56796d95b1f1 WatchSource:0}: Error finding container 7d054730a14ab269983fffe0a33d26143298691554006b7a90ee56796d95b1f1: Status 404 returned error can't find the container with id 7d054730a14ab269983fffe0a33d26143298691554006b7a90ee56796d95b1f1
Sep 11 18:17:03.240817 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:03.240700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v"]
Sep 11 18:17:03.243366 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:17:03.243338    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea3b6a2d_2118_4403_96e4_2e9e4f9eec54.slice/crio-ffb70c6e2bd394c2fd27367f54e50dc6654ec1f7da5ae6c1146d13aab9f39ae8 WatchSource:0}: Error finding container ffb70c6e2bd394c2fd27367f54e50dc6654ec1f7da5ae6c1146d13aab9f39ae8: Status 404 returned error can't find the container with id ffb70c6e2bd394c2fd27367f54e50dc6654ec1f7da5ae6c1146d13aab9f39ae8
Sep 11 18:17:03.533163 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:03.533051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:03.595719 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:03.595684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:03.654372 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:03.654334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:03.793414 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:03.793324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb" event={"ID":"0a73a24f-fb75-42c8-96cb-8e8a809c586d","Type":"ContainerStarted","Data":"7d054730a14ab269983fffe0a33d26143298691554006b7a90ee56796d95b1f1"}
Sep 11 18:17:03.794639 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:03.794610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v" event={"ID":"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54","Type":"ContainerStarted","Data":"ffb70c6e2bd394c2fd27367f54e50dc6654ec1f7da5ae6c1146d13aab9f39ae8"}
Sep 11 18:17:03.928276 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:03.928179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:03.980993 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:03.980956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:05.805009 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:05.804953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb" event={"ID":"0a73a24f-fb75-42c8-96cb-8e8a809c586d","Type":"ContainerStarted","Data":"40ab015bdcbed3975d23b20973784272ec4527fdba9a7a8174cae245448b5a85"}
Sep 11 18:17:05.822189 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:05.822113    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-grkvb" podStartSLOduration=2.314922342 podStartE2EDuration="3.822094118s" totalImagesPullingTime="1.507171776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:05.820496912 +0000 UTC m=+106.917090544" watchObservedRunningTime="2026-09-11 18:17:05.822094118 +0000 UTC m=+106.918687769"
Sep 11 18:17:05.927522 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:05.927464    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:06.132919 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:06.132882    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:07.814610 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:07.814567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v" event={"ID":"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54","Type":"ContainerStarted","Data":"036317fdd10b9743e3301700e472766a79aed226537553d8c3c85bafdc6ed615"}
Sep 11 18:17:07.815010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:07.814622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v" event={"ID":"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54","Type":"ContainerStarted","Data":"6e97999c64f782df9bc54faaef953532fbc3152196267da0904b8f39433c7f18"}
Sep 11 18:17:07.815010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:07.814635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v" event={"ID":"ea3b6a2d-2118-4403-96e4-2e9e4f9eec54","Type":"ContainerStarted","Data":"0943bd3d92254addd840d3e1d3d33cc74ef28ce165a343b817552764fae4b163"}
Sep 11 18:17:07.841144 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:07.841087    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f955f556b-dt78v" podStartSLOduration=1.8029472370000001 podStartE2EDuration="5.841069422s" totalImagesPullingTime="4.038122185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:07.840254564 +0000 UTC m=+108.936848195" watchObservedRunningTime="2026-09-11 18:17:07.841069422 +0000 UTC m=+108.937663052"
Sep 11 18:17:10.228548 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:10.228515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:10.406758 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:10.406710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:10.735878 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:10.735840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:10.786790 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:10.786761    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h87pc"
Sep 11 18:17:11.737401 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:11.737364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:11.800068 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:11.800036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:13.440220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:13.440167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:13.513899 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:13.513866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:13.959631 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:13.959597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:14.150016 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:14.149981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:14.253125 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:14.253055    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:16.575792 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:16.575753    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:16.641659 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:16.641608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:18.166406 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:18.166371    2580 patch_prober.go:28] interesting pod/image-registry-86766dfb79-rqf92 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 18:17:18.166863 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:18.166437    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-86766dfb79-rqf92" podUID="6ff90c73-d339-499d-95a6-b2aa2657f85d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:17:19.344891 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:19.344855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:19.418787 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:19.418751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:19.779816 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:19.779785    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:17:19.987545 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:19.987510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:20.090397 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:20.090309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:23.445356 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:23.445312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:23.545787 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:23.545756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:23.645877 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:23.645840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:24.594456 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.594413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:24.638648 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.638602    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86766dfb79-rqf92"]
Sep 11 18:17:24.731613 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.731578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:24.834920 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.834875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:24.874873 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.874829    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c2ltj"]
Sep 11 18:17:24.877744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.877719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:24.880113 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.880085    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 18:17:24.880339 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.880320    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9m256\""
Sep 11 18:17:24.880477 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.880435    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 18:17:24.880589 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.880578    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 18:17:24.880648 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.880640    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 18:17:24.880828 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.880808    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 18:17:24.880943 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.880926    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 18:17:24.927849 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.927811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09f116e9-3034-4d86-9383-b25b73d9b254-sys\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:24.927849 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.927848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09f116e9-3034-4d86-9383-b25b73d9b254-root\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:24.928105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.927867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-textfile\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:24.928105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.927927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55q2\" (UniqueName: \"kubernetes.io/projected/09f116e9-3034-4d86-9383-b25b73d9b254-kube-api-access-r55q2\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:24.928105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.927946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-tls\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:24.928105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.927970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-wtmp\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:24.928105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.927995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09f116e9-3034-4d86-9383-b25b73d9b254-metrics-client-ca\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:24.928105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.928021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:24.928351 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.928097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:24.934869 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:24.934823    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:25.010938 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.010901    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:25.028947 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.028910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r55q2\" (UniqueName: \"kubernetes.io/projected/09f116e9-3034-4d86-9383-b25b73d9b254-kube-api-access-r55q2\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029078 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.028959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-tls\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029078 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.028986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-wtmp\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029078 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09f116e9-3034-4d86-9383-b25b73d9b254-metrics-client-ca\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029078 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029298 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029298 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09f116e9-3034-4d86-9383-b25b73d9b254-sys\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029298 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-wtmp\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029298 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09f116e9-3034-4d86-9383-b25b73d9b254-root\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029298 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09f116e9-3034-4d86-9383-b25b73d9b254-sys\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029298 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-textfile\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029573 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09f116e9-3034-4d86-9383-b25b73d9b254-root\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029629 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-textfile\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029755 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09f116e9-3034-4d86-9383-b25b73d9b254-metrics-client-ca\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.029816 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.029758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.031505 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.031479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.031576 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.031560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09f116e9-3034-4d86-9383-b25b73d9b254-node-exporter-tls\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.038558 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.038525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55q2\" (UniqueName: \"kubernetes.io/projected/09f116e9-3034-4d86-9383-b25b73d9b254-kube-api-access-r55q2\") pod \"node-exporter-c2ltj\" (UID: \"09f116e9-3034-4d86-9383-b25b73d9b254\") " pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.106126 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.106086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:25.187332 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.187249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c2ltj"
Sep 11 18:17:25.196707 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:17:25.196675    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f116e9_3034_4d86_9383_b25b73d9b254.slice/crio-10f03627636adbcec3bbdc1922687827649724cc13042aea39c1ab941e256cf7 WatchSource:0}: Error finding container 10f03627636adbcec3bbdc1922687827649724cc13042aea39c1ab941e256cf7: Status 404 returned error can't find the container with id 10f03627636adbcec3bbdc1922687827649724cc13042aea39c1ab941e256cf7
Sep 11 18:17:25.205924 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.205886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:25.305239 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.305185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:25.364872 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.364838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:25.680715 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.680627    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:25.780714 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.780672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:25.875884 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.875846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2ltj" event={"ID":"09f116e9-3034-4d86-9383-b25b73d9b254","Type":"ContainerStarted","Data":"10f03627636adbcec3bbdc1922687827649724cc13042aea39c1ab941e256cf7"}
Sep 11 18:17:25.880861 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.880824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:25.979222 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:25.979065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:26.225950 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.225915    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:26.880485 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.880450    2580 generic.go:356] "Generic (PLEG): container finished" podID="09f116e9-3034-4d86-9383-b25b73d9b254" containerID="1db1287ee5ce04009068913272643a8936b6bc2c1d4755295246ea36266a9bbb" exitCode=0
Sep 11 18:17:26.880865 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.880529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2ltj" event={"ID":"09f116e9-3034-4d86-9383-b25b73d9b254","Type":"ContainerDied","Data":"1db1287ee5ce04009068913272643a8936b6bc2c1d4755295246ea36266a9bbb"}
Sep 11 18:17:26.959340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.959312    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-857594858f-cvgsj"]
Sep 11 18:17:26.962661 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.962638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:26.965752 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.965727    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 18:17:26.965752 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.965745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 18:17:26.965956 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.965727    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 18:17:26.965956 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.965784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 18:17:26.965956 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.965748    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 18:17:26.966102 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.966080    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4w6hj\""
Sep 11 18:17:26.966222 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.966183    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6d6367s9qmo10\""
Sep 11 18:17:26.979730 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:26.979695    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-857594858f-cvgsj"]
Sep 11 18:17:27.045425 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.045376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.045612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.045444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxg5x\" (UniqueName: \"kubernetes.io/projected/9932a593-5517-4a43-b4be-952219bb1435-kube-api-access-wxg5x\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.045612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.045470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9932a593-5517-4a43-b4be-952219bb1435-metrics-client-ca\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.045612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.045485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.045612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.045503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.045612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.045520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.045612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.045543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-tls\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.045612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.045598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-grpc-tls\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.146413 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.146319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.146413 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.146377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxg5x\" (UniqueName: \"kubernetes.io/projected/9932a593-5517-4a43-b4be-952219bb1435-kube-api-access-wxg5x\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.146413 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.146399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9932a593-5517-4a43-b4be-952219bb1435-metrics-client-ca\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.146690 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.146425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.146690 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.146442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.146690 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.146469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.146690 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.146505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-tls\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.146690 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.146546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-grpc-tls\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.147362 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.147329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9932a593-5517-4a43-b4be-952219bb1435-metrics-client-ca\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.149226 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.149177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-tls\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.149524 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.149359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-grpc-tls\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.149524 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.149459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.149646 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.149561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.149646 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.149576    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.149901 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.149882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9932a593-5517-4a43-b4be-952219bb1435-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.156040 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.156014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxg5x\" (UniqueName: \"kubernetes.io/projected/9932a593-5517-4a43-b4be-952219bb1435-kube-api-access-wxg5x\") pod \"thanos-querier-857594858f-cvgsj\" (UID: \"9932a593-5517-4a43-b4be-952219bb1435\") " pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.293542 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.293495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:27.570606 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.570521    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-857594858f-cvgsj"]
Sep 11 18:17:27.573695 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:17:27.573668    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9932a593_5517_4a43_b4be_952219bb1435.slice/crio-e8379b185ab56784d136f2e753280259768b763f924fe1b0e30cbd185acb2b1a WatchSource:0}: Error finding container e8379b185ab56784d136f2e753280259768b763f924fe1b0e30cbd185acb2b1a: Status 404 returned error can't find the container with id e8379b185ab56784d136f2e753280259768b763f924fe1b0e30cbd185acb2b1a
Sep 11 18:17:27.863066 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.862977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:27.888864 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.888827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj" event={"ID":"9932a593-5517-4a43-b4be-952219bb1435","Type":"ContainerStarted","Data":"e8379b185ab56784d136f2e753280259768b763f924fe1b0e30cbd185acb2b1a"}
Sep 11 18:17:27.890996 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.890969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2ltj" event={"ID":"09f116e9-3034-4d86-9383-b25b73d9b254","Type":"ContainerStarted","Data":"72f971992b70cf9b2d90b4e0b80f75975aeaa6c437b7217381fe3c9dc5f86c67"}
Sep 11 18:17:27.891123 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.891025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2ltj" event={"ID":"09f116e9-3034-4d86-9383-b25b73d9b254","Type":"ContainerStarted","Data":"0326f4590bd0bb8b10d73e2dfd22f192956de1077575d050c56478d863438c80"}
Sep 11 18:17:27.910495 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.910440    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c2ltj" podStartSLOduration=2.9857698900000003 podStartE2EDuration="3.910423456s" totalImagesPullingTime="859.676743ms" totalInitContainerRuntime="64.976823ms" isStatefulPod=true podCreationTimestamp="2026-09-11 18:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:27.910366153 +0000 UTC m=+129.006959797" watchObservedRunningTime="2026-09-11 18:17:27.910423456 +0000 UTC m=+129.007017088"
Sep 11 18:17:27.963488 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:27.963449    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:28.133919 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:28.133884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:28.336334 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:28.336300    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:29.015315 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.015273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:29.163379 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.163339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:17:29.165646 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.165621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53-metrics-certs\") pod \"network-metrics-daemon-plhmr\" (UID: \"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53\") " pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:17:29.313299 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.313183    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28rql\""
Sep 11 18:17:29.322104 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.322070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plhmr"
Sep 11 18:17:29.616825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.616711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:29.683359 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.683326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:29.885782 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.885745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:29.901065 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.901033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj" event={"ID":"9932a593-5517-4a43-b4be-952219bb1435","Type":"ContainerStarted","Data":"7a061aa47697c1bc27f7b46c7beed634698deba537b46130948d0b87ca05c72b"}
Sep 11 18:17:29.901241 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.901103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj" event={"ID":"9932a593-5517-4a43-b4be-952219bb1435","Type":"ContainerStarted","Data":"e3ed9bbb7127d5754a5882750dbc839fcbb5e386c6b3fd576282f9f0b129a9fd"}
Sep 11 18:17:29.953926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:29.953893    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-plhmr"]
Sep 11 18:17:29.956448 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:17:29.956419    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd47ada5_8d61_47d9_a7d5_eeab0b2b2b53.slice/crio-aafb0728707baca6bcd9ca95eca005830808193bbe2cfb6128bf3b6484f1a5fe WatchSource:0}: Error finding container aafb0728707baca6bcd9ca95eca005830808193bbe2cfb6128bf3b6484f1a5fe: Status 404 returned error can't find the container with id aafb0728707baca6bcd9ca95eca005830808193bbe2cfb6128bf3b6484f1a5fe
Sep 11 18:17:30.826667 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:30.826549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:30.907363 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:30.907331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plhmr" event={"ID":"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53","Type":"ContainerStarted","Data":"aafb0728707baca6bcd9ca95eca005830808193bbe2cfb6128bf3b6484f1a5fe"}
Sep 11 18:17:30.910575 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:30.910514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj" event={"ID":"9932a593-5517-4a43-b4be-952219bb1435","Type":"ContainerStarted","Data":"89e844198292ff9a100adec36838ff41cfa2988bc256560545086b208d93d0d5"}
Sep 11 18:17:30.910714 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:30.910615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj" event={"ID":"9932a593-5517-4a43-b4be-952219bb1435","Type":"ContainerStarted","Data":"d5774a5bbdb29fd9c7594ba096ec8abfa7c083964dc9b10227d15ac4f2ba7cff"}
Sep 11 18:17:30.926829 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:30.926797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:31.241268 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:31.241228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:31.917596 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:31.917546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj" event={"ID":"9932a593-5517-4a43-b4be-952219bb1435","Type":"ContainerStarted","Data":"3f71ec35c507e4e257bf51b30e8984ef87aca6c2cf15fbe7c1141dfbe3b8cfd7"}
Sep 11 18:17:31.918083 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:31.917624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj" event={"ID":"9932a593-5517-4a43-b4be-952219bb1435","Type":"ContainerStarted","Data":"5a097c3c2dbbe74adc0ef35a536a1bc32e9205a8232794b4570eb68f730c4376"}
Sep 11 18:17:31.918083 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:31.917693    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:31.919383 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:31.919355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plhmr" event={"ID":"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53","Type":"ContainerStarted","Data":"b05e2603808feebeb3e00053d86fae67e444a8709a9251218e71d3a1dd3da450"}
Sep 11 18:17:31.919509 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:31.919409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plhmr" event={"ID":"cd47ada5-8d61-47d9-a7d5-eeab0b2b2b53","Type":"ContainerStarted","Data":"f53476d09b4f41143ab5da213dac08f4ef4a646cac924485c3aec6ccceef9dd4"}
Sep 11 18:17:31.941969 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:31.941898    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj" podStartSLOduration=2.928638478 podStartE2EDuration="5.941883478s" totalImagesPullingTime="3.013245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:31.940816557 +0000 UTC m=+133.037410187" watchObservedRunningTime="2026-09-11 18:17:31.941883478 +0000 UTC m=+133.038477153"
Sep 11 18:17:31.957654 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:31.957596    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-plhmr" podStartSLOduration=131.706343258 podStartE2EDuration="2m12.957581276s" totalImagesPullingTime="1.251238018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:31.95685198 +0000 UTC m=+133.053445612" watchObservedRunningTime="2026-09-11 18:17:31.957581276 +0000 UTC m=+133.054174906"
Sep 11 18:17:32.058711 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:32.058666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:32.160633 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:32.160596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:32.218603 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:32.218524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:34.742045 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:34.742008    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:35.544211 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:35.544159    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:36.596890 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:36.596853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:36.693750 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:36.693714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:36.792869 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:36.792828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:37.928966 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:37.928938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-857594858f-cvgsj"
Sep 11 18:17:39.161190 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:39.161139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:39.269340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:39.269301    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:44.919958 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:44.919913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:44.979556 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:44.979517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:45.040611 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:45.040567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:45.550972 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:45.550938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:47.111248 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:47.111216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:47.167978 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:47.167943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:49.661591 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.661530    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-86766dfb79-rqf92" podUID="6ff90c73-d339-499d-95a6-b2aa2657f85d" containerName="registry" containerID="cri-o://548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8" gracePeriod=30
Sep 11 18:17:49.776680 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.776636    2580 patch_prober.go:28] interesting pod/image-registry-86766dfb79-rqf92 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.133.0.6:5000/healthz\": dial tcp 10.133.0.6:5000: connect: connection refused" start-of-body=
Sep 11 18:17:49.776820 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.776712    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-86766dfb79-rqf92" podUID="6ff90c73-d339-499d-95a6-b2aa2657f85d" containerName="registry" probeResult="failure" output="Get \"https://10.133.0.6:5000/healthz\": dial tcp 10.133.0.6:5000: connect: connection refused"
Sep 11 18:17:49.900415 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.900384    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:17:49.981660 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.981569    2580 generic.go:356] "Generic (PLEG): container finished" podID="6ff90c73-d339-499d-95a6-b2aa2657f85d" containerID="548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8" exitCode=0
Sep 11 18:17:49.981660 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.981616    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86766dfb79-rqf92"
Sep 11 18:17:49.981660 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.981629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86766dfb79-rqf92" event={"ID":"6ff90c73-d339-499d-95a6-b2aa2657f85d","Type":"ContainerDied","Data":"548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8"}
Sep 11 18:17:49.981882 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.981676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86766dfb79-rqf92" event={"ID":"6ff90c73-d339-499d-95a6-b2aa2657f85d","Type":"ContainerDied","Data":"34a36c72c92f0819efa87efe8484491cba1c443d8098cae41499d2eae14eede2"}
Sep 11 18:17:49.981882 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.981697    2580 scope.go:117] "RemoveContainer" containerID="548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8"
Sep 11 18:17:49.990087 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.990067    2580 scope.go:117] "RemoveContainer" containerID="548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8"
Sep 11 18:17:49.990396 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:17:49.990378    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8\": container with ID starting with 548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8 not found: ID does not exist" containerID="548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8"
Sep 11 18:17:49.990451 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:49.990407    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8"} err="failed to get container status \"548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8\": rpc error: code = NotFound desc = could not find container \"548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8\": container with ID starting with 548d85809bcadfb3e52ad5547db72003f20015dc9484f32fd6bd2eb524045cd8 not found: ID does not exist"
Sep 11 18:17:50.023029 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.022992    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-trusted-ca\") pod \"6ff90c73-d339-499d-95a6-b2aa2657f85d\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") "
Sep 11 18:17:50.023120 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.023042    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-bound-sa-token\") pod \"6ff90c73-d339-499d-95a6-b2aa2657f85d\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") "
Sep 11 18:17:50.023120 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.023063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znbrq\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-kube-api-access-znbrq\") pod \"6ff90c73-d339-499d-95a6-b2aa2657f85d\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") "
Sep 11 18:17:50.023120 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.023091    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-image-registry-private-configuration\") pod \"6ff90c73-d339-499d-95a6-b2aa2657f85d\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") "
Sep 11 18:17:50.023120 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.023108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-installation-pull-secrets\") pod \"6ff90c73-d339-499d-95a6-b2aa2657f85d\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") "
Sep 11 18:17:50.023324 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.023133    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") pod \"6ff90c73-d339-499d-95a6-b2aa2657f85d\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") "
Sep 11 18:17:50.023324 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.023169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-certificates\") pod \"6ff90c73-d339-499d-95a6-b2aa2657f85d\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") "
Sep 11 18:17:50.023324 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.023238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ff90c73-d339-499d-95a6-b2aa2657f85d-ca-trust-extracted\") pod \"6ff90c73-d339-499d-95a6-b2aa2657f85d\" (UID: \"6ff90c73-d339-499d-95a6-b2aa2657f85d\") "
Sep 11 18:17:50.023682 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.023621    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6ff90c73-d339-499d-95a6-b2aa2657f85d" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:17:50.023814 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.023719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6ff90c73-d339-499d-95a6-b2aa2657f85d" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:17:50.025599 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.025562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6ff90c73-d339-499d-95a6-b2aa2657f85d" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:17:50.025883 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.025854    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6ff90c73-d339-499d-95a6-b2aa2657f85d" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:17:50.025980 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.025931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-kube-api-access-znbrq" (OuterVolumeSpecName: "kube-api-access-znbrq") pod "6ff90c73-d339-499d-95a6-b2aa2657f85d" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d"). InnerVolumeSpecName "kube-api-access-znbrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:17:50.025980 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.025944    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6ff90c73-d339-499d-95a6-b2aa2657f85d" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:17:50.025980 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.025963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6ff90c73-d339-499d-95a6-b2aa2657f85d" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:17:50.031856 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.031829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff90c73-d339-499d-95a6-b2aa2657f85d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6ff90c73-d339-499d-95a6-b2aa2657f85d" (UID: "6ff90c73-d339-499d-95a6-b2aa2657f85d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:17:50.124409 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.124361    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-trusted-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:17:50.124409 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.124403    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-bound-sa-token\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:17:50.124409 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.124413    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znbrq\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-kube-api-access-znbrq\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:17:50.124409 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.124423    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-image-registry-private-configuration\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:17:50.124652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.124433    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ff90c73-d339-499d-95a6-b2aa2657f85d-installation-pull-secrets\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:17:50.124652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.124442    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:17:50.124652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.124450    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ff90c73-d339-499d-95a6-b2aa2657f85d-registry-certificates\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:17:50.124652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.124459    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ff90c73-d339-499d-95a6-b2aa2657f85d-ca-trust-extracted\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:17:50.304479 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.304439    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86766dfb79-rqf92"]
Sep 11 18:17:50.310576 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:50.310548    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86766dfb79-rqf92"]
Sep 11 18:17:51.350988 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:51.350949    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:51.394495 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:51.394461    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff90c73-d339-499d-95a6-b2aa2657f85d" path="/var/lib/kubelet/pods/6ff90c73-d339-499d-95a6-b2aa2657f85d/volumes"
Sep 11 18:17:51.451724 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:51.451691    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:17:51.552291 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:17:51.552259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:02.130267 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:02.130224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:02.201003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:02.200962    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:23.967074 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:23.967033    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:44.177559 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:44.177510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:44.386947 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:44.386915    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:51.958817 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:51.958767    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:52.085356 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:52.085324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:52.147935 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:52.147903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:52.741698 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:52.741659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:52.809314 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:52.809286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:59.496878 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:59.496844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:18:59.597845 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:18:59.597812    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:02.030884 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:02.030846    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:02.091097 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:02.091062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:02.638983 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:02.638946    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:02.695811 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:02.695774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:11.334618 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:11.334579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:11.432680 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:11.432640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:12.283030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:12.282409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:12.383278 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:12.383242    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:12.442860 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:12.442825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:17.129930 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:17.129890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:17.230986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:17.230941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:32.027895 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:32.027856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:32.082357 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:32.082320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:19:46.210783 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:19:46.210693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:10.944043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:10.944001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:11.041730 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:11.041688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:19.286726 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:19.286689    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 18:20:37.402172 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:37.402125    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:38.766054 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:38.766019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:41.025374 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:41.025340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:41.125976 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:41.125936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:42.576052 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:42.576018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:42.676766 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:42.676725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:42.941360 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:42.941324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:43.043670 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:43.043631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:44.775469 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:44.775435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:44.958981 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:44.958945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:45.193300 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.193263    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"]
Sep 11 18:20:45.193555 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.193543    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff90c73-d339-499d-95a6-b2aa2657f85d" containerName="registry"
Sep 11 18:20:45.193601 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.193558    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff90c73-d339-499d-95a6-b2aa2657f85d" containerName="registry"
Sep 11 18:20:45.193635 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.193612    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff90c73-d339-499d-95a6-b2aa2657f85d" containerName="registry"
Sep 11 18:20:45.196322 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.196303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"
Sep 11 18:20:45.198387 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.198366    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 18:20:45.198506 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.198484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4p4z5\""
Sep 11 18:20:45.199092 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.199073    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 18:20:45.206243 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.206216    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"]
Sep 11 18:20:45.289118 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.289081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-w4tjr\" (UID: \"cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"
Sep 11 18:20:45.289326 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.289138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wvc\" (UniqueName: \"kubernetes.io/projected/cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0-kube-api-access-47wvc\") pod \"cert-manager-cainjector-559cdd8c87-w4tjr\" (UID: \"cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"
Sep 11 18:20:45.390038 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.389995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wvc\" (UniqueName: \"kubernetes.io/projected/cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0-kube-api-access-47wvc\") pod \"cert-manager-cainjector-559cdd8c87-w4tjr\" (UID: \"cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"
Sep 11 18:20:45.390267 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.390081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-w4tjr\" (UID: \"cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"
Sep 11 18:20:45.400171 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.400135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-w4tjr\" (UID: \"cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"
Sep 11 18:20:45.400171 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.400162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wvc\" (UniqueName: \"kubernetes.io/projected/cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0-kube-api-access-47wvc\") pod \"cert-manager-cainjector-559cdd8c87-w4tjr\" (UID: \"cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"
Sep 11 18:20:45.505346 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.505254    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"
Sep 11 18:20:45.776164 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.776084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr"]
Sep 11 18:20:45.779212 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:20:45.779173    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcecc0a4b_2a1e_4977_b4dd_fe67cd5502d0.slice/crio-7dd48e04596dd91e1d0becad8012be5ae8636f87cbe00d019e80244049dbdc25 WatchSource:0}: Error finding container 7dd48e04596dd91e1d0becad8012be5ae8636f87cbe00d019e80244049dbdc25: Status 404 returned error can't find the container with id 7dd48e04596dd91e1d0becad8012be5ae8636f87cbe00d019e80244049dbdc25
Sep 11 18:20:45.781278 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:45.781259    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:20:46.623087 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:46.623025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr" event={"ID":"cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0","Type":"ContainerStarted","Data":"7dd48e04596dd91e1d0becad8012be5ae8636f87cbe00d019e80244049dbdc25"}
Sep 11 18:20:47.813467 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:47.813417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:47.915052 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:47.914994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:49.492263 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:49.492218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:49.560299 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:49.560262    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:49.636111 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:49.636069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr" event={"ID":"cecc0a4b-2a1e-4977-b4dd-fe67cd5502d0","Type":"ContainerStarted","Data":"8803f4b6a75c933ad0869e16a39f3ab20d359936e0e237d9be0d896f1931f8bd"}
Sep 11 18:20:49.656236 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:49.656165    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w4tjr" podStartSLOduration=1.722044881 podStartE2EDuration="4.656147972s" totalImagesPullingTime="2.934103091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:49.654941848 +0000 UTC m=+330.751535476" watchObservedRunningTime="2026-09-11 18:20:49.656147972 +0000 UTC m=+330.752741602"
Sep 11 18:20:52.784445 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:52.784404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:20:53.321967 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.321926    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j9df9"]
Sep 11 18:20:53.325082 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.325062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-j9df9"
Sep 11 18:20:53.327106 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.327087    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9sdv6\""
Sep 11 18:20:53.334794 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.334765    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j9df9"]
Sep 11 18:20:53.450893 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.450855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6-bound-sa-token\") pod \"cert-manager-55b4d954d5-j9df9\" (UID: \"9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6\") " pod="cert-manager/cert-manager-55b4d954d5-j9df9"
Sep 11 18:20:53.450893 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.450892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrsj\" (UniqueName: \"kubernetes.io/projected/9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6-kube-api-access-bhrsj\") pod \"cert-manager-55b4d954d5-j9df9\" (UID: \"9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6\") " pod="cert-manager/cert-manager-55b4d954d5-j9df9"
Sep 11 18:20:53.551997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.551950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6-bound-sa-token\") pod \"cert-manager-55b4d954d5-j9df9\" (UID: \"9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6\") " pod="cert-manager/cert-manager-55b4d954d5-j9df9"
Sep 11 18:20:53.551997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.551995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrsj\" (UniqueName: \"kubernetes.io/projected/9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6-kube-api-access-bhrsj\") pod \"cert-manager-55b4d954d5-j9df9\" (UID: \"9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6\") " pod="cert-manager/cert-manager-55b4d954d5-j9df9"
Sep 11 18:20:53.562363 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.562328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6-bound-sa-token\") pod \"cert-manager-55b4d954d5-j9df9\" (UID: \"9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6\") " pod="cert-manager/cert-manager-55b4d954d5-j9df9"
Sep 11 18:20:53.562513 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.562414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrsj\" (UniqueName: \"kubernetes.io/projected/9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6-kube-api-access-bhrsj\") pod \"cert-manager-55b4d954d5-j9df9\" (UID: \"9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6\") " pod="cert-manager/cert-manager-55b4d954d5-j9df9"
Sep 11 18:20:53.634376 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.634329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-j9df9"
Sep 11 18:20:53.904420 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:53.904318    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j9df9"]
Sep 11 18:20:53.908784 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:20:53.908752    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b5b7ee6_b3e3_4d67_89db_3fa5033e92a6.slice/crio-3f7098922d5eb3b98804aa012cd0c36837f252bee0b8576fda3951ddc4b1087d WatchSource:0}: Error finding container 3f7098922d5eb3b98804aa012cd0c36837f252bee0b8576fda3951ddc4b1087d: Status 404 returned error can't find the container with id 3f7098922d5eb3b98804aa012cd0c36837f252bee0b8576fda3951ddc4b1087d
Sep 11 18:20:54.657189 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:54.657153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-j9df9" event={"ID":"9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6","Type":"ContainerStarted","Data":"29a5e27554b4ea895e5079aacddd4125925b17da185dbc26e2398f58d42769df"}
Sep 11 18:20:54.657189 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:54.657207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-j9df9" event={"ID":"9b5b7ee6-b3e3-4d67-89db-3fa5033e92a6","Type":"ContainerStarted","Data":"3f7098922d5eb3b98804aa012cd0c36837f252bee0b8576fda3951ddc4b1087d"}
Sep 11 18:20:54.674340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:54.674258    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-j9df9" podStartSLOduration=1.67423732 podStartE2EDuration="1.67423732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:54.673518307 +0000 UTC m=+335.770111937" watchObservedRunningTime="2026-09-11 18:20:54.67423732 +0000 UTC m=+335.770830952"
Sep 11 18:20:59.248176 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:20:59.248134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:00.817493 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:00.817456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:05.261525 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:05.261485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:05.361707 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:05.361669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:07.451563 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:07.451512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:17.764645 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.764559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:17.844265 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.844225    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"]
Sep 11 18:21:17.847180 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.847160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:17.850901 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.850871    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 18:21:17.851030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.850911    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 18:21:17.851184 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.851170    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:21:17.851881 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.851860    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 18:21:17.851987 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.851864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kq4q9\""
Sep 11 18:21:17.852048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.852002    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 18:21:17.860713 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.860682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"]
Sep 11 18:21:17.872802 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.872765    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:17.930554 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.930514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-metrics-cert\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:17.930904 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.930877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-manager-config\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:17.931032 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.930913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4d5h\" (UniqueName: \"kubernetes.io/projected/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-kube-api-access-f4d5h\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:17.931032 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:17.930942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-cert\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:18.031408 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.031311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-manager-config\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:18.031408 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.031355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4d5h\" (UniqueName: \"kubernetes.io/projected/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-kube-api-access-f4d5h\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:18.031408 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.031378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-cert\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:18.031636 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.031422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-metrics-cert\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:18.031970 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.031939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-manager-config\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:18.033880 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.033858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-metrics-cert\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:18.033963 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.033937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-cert\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:18.040450 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.040417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4d5h\" (UniqueName: \"kubernetes.io/projected/6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15-kube-api-access-f4d5h\") pod \"lws-controller-manager-5588b8499b-v8pc7\" (UID: \"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:18.156345 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.156302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:18.434124 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.434047    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"]
Sep 11 18:21:18.438496 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:21:18.438465    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2df4b0_974f_43c7_a6b9_1fa3bb9c1a15.slice/crio-f28407ec4eb111c7e0cf43f723c6a936210ebb115201c6848ef096216f86291c WatchSource:0}: Error finding container f28407ec4eb111c7e0cf43f723c6a936210ebb115201c6848ef096216f86291c: Status 404 returned error can't find the container with id f28407ec4eb111c7e0cf43f723c6a936210ebb115201c6848ef096216f86291c
Sep 11 18:21:18.750271 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:18.750162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7" event={"ID":"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15","Type":"ContainerStarted","Data":"f28407ec4eb111c7e0cf43f723c6a936210ebb115201c6848ef096216f86291c"}
Sep 11 18:21:19.733418 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:19.733378    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:19.833193 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:19.833160    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:20.386170 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.386108    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:20.673695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.673591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:20.778909 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.778874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:20.853443 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.850367    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"]
Sep 11 18:21:20.855273 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.855243    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:20.857639 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.857619    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 18:21:20.857757 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.857621    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8v878\""
Sep 11 18:21:20.857827 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.857808    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 18:21:20.857880 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.857820    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 18:21:20.857986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.857965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 18:21:20.866478 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.866452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"]
Sep 11 18:21:20.954150 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.954062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/362492d6-442f-48c8-a982-003efc62c937-tmp\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:20.954150 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.954098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/362492d6-442f-48c8-a982-003efc62c937-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:20.954150 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.954135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/362492d6-442f-48c8-a982-003efc62c937-manifests\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:20.954385 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.954234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/362492d6-442f-48c8-a982-003efc62c937-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:20.954385 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:20.954300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq664\" (UniqueName: \"kubernetes.io/projected/362492d6-442f-48c8-a982-003efc62c937-kube-api-access-tq664\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.002273 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.002224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:21.054715 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.054671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/362492d6-442f-48c8-a982-003efc62c937-manifests\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.054830 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.054767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/362492d6-442f-48c8-a982-003efc62c937-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.054830 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.054803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq664\" (UniqueName: \"kubernetes.io/projected/362492d6-442f-48c8-a982-003efc62c937-kube-api-access-tq664\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.054931 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.054828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/362492d6-442f-48c8-a982-003efc62c937-tmp\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.054931 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.054851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/362492d6-442f-48c8-a982-003efc62c937-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.055163 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.055133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/362492d6-442f-48c8-a982-003efc62c937-manifests\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.057103 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.057075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/362492d6-442f-48c8-a982-003efc62c937-tmp\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.057403 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.057382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/362492d6-442f-48c8-a982-003efc62c937-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.057479 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.057396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/362492d6-442f-48c8-a982-003efc62c937-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.064545 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.064518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq664\" (UniqueName: \"kubernetes.io/projected/362492d6-442f-48c8-a982-003efc62c937-kube-api-access-tq664\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-ml5cm\" (UID: \"362492d6-442f-48c8-a982-003efc62c937\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.103606 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.103566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:21.173729 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.173685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:21.202518 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.202472    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:21.302749 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.302710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:21.406266 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.406227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:21.452803 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.452768    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"]
Sep 11 18:21:21.456091 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:21:21.456063    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362492d6_442f_48c8_a982_003efc62c937.slice/crio-21a2abb752412f37972300362e4db9dd3ad82d705d09ccf858413a13b20438c8 WatchSource:0}: Error finding container 21a2abb752412f37972300362e4db9dd3ad82d705d09ccf858413a13b20438c8: Status 404 returned error can't find the container with id 21a2abb752412f37972300362e4db9dd3ad82d705d09ccf858413a13b20438c8
Sep 11 18:21:21.764827 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.764790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm" event={"ID":"362492d6-442f-48c8-a982-003efc62c937","Type":"ContainerStarted","Data":"21a2abb752412f37972300362e4db9dd3ad82d705d09ccf858413a13b20438c8"}
Sep 11 18:21:21.766305 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.766275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7" event={"ID":"6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15","Type":"ContainerStarted","Data":"bcd3fefde8fafe51bb5f078ace00b601f7f66bd02329eaa222cba57ce9d721d3"}
Sep 11 18:21:21.766529 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.766509    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:21.783326 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:21.783274    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7" podStartSLOduration=2.377519286 podStartE2EDuration="4.783253055s" totalImagesPullingTime="2.405733769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:21.782664528 +0000 UTC m=+362.879258154" watchObservedRunningTime="2026-09-11 18:21:21.783253055 +0000 UTC m=+362.879846684"
Sep 11 18:21:24.781615 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:24.781576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm" event={"ID":"362492d6-442f-48c8-a982-003efc62c937","Type":"ContainerStarted","Data":"2010e2cfcaa06483714b2af6e6cbabafed2de77a89b78e0b83bdb08f7110aa23"}
Sep 11 18:21:24.831359 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:24.831318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:24.924851 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:24.924817    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:25.787574 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:25.787540    2580 generic.go:356] "Generic (PLEG): container finished" podID="362492d6-442f-48c8-a982-003efc62c937" containerID="2010e2cfcaa06483714b2af6e6cbabafed2de77a89b78e0b83bdb08f7110aa23" exitCode=0
Sep 11 18:21:25.787961 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:25.787578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm" event={"ID":"362492d6-442f-48c8-a982-003efc62c937","Type":"ContainerDied","Data":"2010e2cfcaa06483714b2af6e6cbabafed2de77a89b78e0b83bdb08f7110aa23"}
Sep 11 18:21:26.800284 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:26.800241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm" event={"ID":"362492d6-442f-48c8-a982-003efc62c937","Type":"ContainerStarted","Data":"b8b178c84c40caf1b6ce215ff7b297972cb5883331955917be8e3d1fa48cc10b"}
Sep 11 18:21:26.800680 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:26.800380    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:26.835165 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:26.835102    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm" podStartSLOduration=2.166464348 podStartE2EDuration="6.835082915s" totalImagesPullingTime="3.581523464s" totalInitContainerRuntime="1.087095103s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:26.833359147 +0000 UTC m=+367.929952777" watchObservedRunningTime="2026-09-11 18:21:26.835082915 +0000 UTC m=+367.931676545"
Sep 11 18:21:31.182240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:31.182179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:31.280745 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:31.280698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:32.773831 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:32.773789    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-v8pc7"
Sep 11 18:21:37.728959 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:37.728915    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:37.789020 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:37.788983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:37.807372 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:37.807341    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-ml5cm"
Sep 11 18:21:38.161842 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:38.161809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:38.227686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:38.227649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:40.142969 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:40.142927    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:40.257407 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:40.257369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:50.822774 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:50.822733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:50.925538 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:50.925488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:51.284460 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:51.284411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:51.378252 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:51.378188    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:51.851073 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:51.851033    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:52.027575 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:52.027540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:52.118072 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:52.117979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:52.220487 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:52.220451    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:52.516313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:52.516273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:53.043747 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:53.043700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:56.561889 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:56.561848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:56.661714 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:56.661677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:58.422474 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.422431    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"]
Sep 11 18:21:58.436870 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.436826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.438754 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.438718    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"]
Sep 11 18:21:58.439566 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.439540    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 18:21:58.439703 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.439585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 18:21:58.439703 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.439604    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2gzbm\""
Sep 11 18:21:58.439937 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.439910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 18:21:58.439937 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.439926    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 18:21:58.440042 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.439964    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 18:21:58.440092 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.439911    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 18:21:58.536954 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.536910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90d8bfb8-bff1-4ad4-8707-938e18139c93-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.537163 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.536966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.537163 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.536996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5gs\" (UniqueName: \"kubernetes.io/projected/90d8bfb8-bff1-4ad4-8707-938e18139c93-kube-api-access-bt5gs\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.537163 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.537030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.537163 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.537093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.537163 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.537141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.537401 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.537192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d8bfb8-bff1-4ad4-8707-938e18139c93-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.638421 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.638376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.638421 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.638419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5gs\" (UniqueName: \"kubernetes.io/projected/90d8bfb8-bff1-4ad4-8707-938e18139c93-kube-api-access-bt5gs\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.638651 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.638450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.638651 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.638471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.638651 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.638489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.638651 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.638511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d8bfb8-bff1-4ad4-8707-938e18139c93-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.638651 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.638562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90d8bfb8-bff1-4ad4-8707-938e18139c93-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.639271 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.639242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.641005 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.640973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90d8bfb8-bff1-4ad4-8707-938e18139c93-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.641107 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.641027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d8bfb8-bff1-4ad4-8707-938e18139c93-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.641107 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.641050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.641175 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.641117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.650502 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.650477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d8bfb8-bff1-4ad4-8707-938e18139c93-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.651296 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.651265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5gs\" (UniqueName: \"kubernetes.io/projected/90d8bfb8-bff1-4ad4-8707-938e18139c93-kube-api-access-bt5gs\") pod \"istiod-openshift-gateway-5677ccb49b-4mz5c\" (UID: \"90d8bfb8-bff1-4ad4-8707-938e18139c93\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.751825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.751727    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:21:58.897245 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:58.897192    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:59.000503 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:59.000469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:59.052088 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:59.052012    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"]
Sep 11 18:21:59.055477 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:21:59.055445    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d8bfb8_bff1_4ad4_8707_938e18139c93.slice/crio-fbb1468aa5ae3d3dec3880049b348db15a18dab404bc760efba13a2cf8d3f8de WatchSource:0}: Error finding container fbb1468aa5ae3d3dec3880049b348db15a18dab404bc760efba13a2cf8d3f8de: Status 404 returned error can't find the container with id fbb1468aa5ae3d3dec3880049b348db15a18dab404bc760efba13a2cf8d3f8de
Sep 11 18:21:59.747788 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:59.747745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:21:59.935569 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:59.935499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c" event={"ID":"90d8bfb8-bff1-4ad4-8707-938e18139c93","Type":"ContainerStarted","Data":"fbb1468aa5ae3d3dec3880049b348db15a18dab404bc760efba13a2cf8d3f8de"}
Sep 11 18:21:59.959235 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:21:59.959102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:00.158731 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:00.158682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:01.249649 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:01.249607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:01.339405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:01.339370    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:01.488004 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:01.487948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:01.598333 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:01.598273    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:22:01.598466 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:01.598373    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:22:01.678109 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:01.678071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:01.777521 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:01.777477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:01.946069 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:01.946034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c" event={"ID":"90d8bfb8-bff1-4ad4-8707-938e18139c93","Type":"ContainerStarted","Data":"7a361f20584cad96571918e553660f9933474d1d0101af895ff63c64795e07f6"}
Sep 11 18:22:01.946289 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:01.946224    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:22:01.975034 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:01.974969    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c" podStartSLOduration=1.434659763 podStartE2EDuration="3.974946158s" totalImagesPullingTime="2.540286395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:01.972284621 +0000 UTC m=+403.068878253" watchObservedRunningTime="2026-09-11 18:22:01.974946158 +0000 UTC m=+403.071539790"
Sep 11 18:22:02.952119 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:02.952090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4mz5c"
Sep 11 18:22:13.141970 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:13.141932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:13.242208 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:13.242163    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:14.207969 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:14.207930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:24.205779 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:24.205736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:24.264499 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:24.264455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:24.333816 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:24.333777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:38.987615 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:38.987570    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:52.311029 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:52.310904    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:22:52.411105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:22:52.411071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:05.744628 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:05.744586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:08.175971 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:08.175912    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:08.551489 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:08.551353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:10.606888 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:10.606828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:10.745458 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:10.745395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:11.349096 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:11.349060    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:11.739159 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:11.739120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:12.793396 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:12.793355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:13.801230 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:13.801180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:13.953688 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:13.953644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:15.150680 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:15.150636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:15.887170 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:15.887138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:18.158475 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:18.158422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:18.262508 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:18.262470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:21.960502 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:21.960430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:22.082976 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:22.082939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:22.183519 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:22.183480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:22.239527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:22.239435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:22.547881 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:22.547787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:22.648582 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:22.648541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:27.500679 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:27.500640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:27.601087 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:27.601048    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:27.708882 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:27.708848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:34.068799 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:34.068755    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:34.166818 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:34.166779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:38.386667 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:38.386625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:38.487091 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:38.487044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:42.185717 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:42.185679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:42.241724 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:42.241675    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:42.387129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:42.387082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:42.455694 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:42.455605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:47.780650 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:47.780602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:47.881665 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:47.881617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:47.981437 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:47.981403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:48.042330 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:48.042225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:48.183867 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:48.183830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:48.240743 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:48.240709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:54.084618 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:54.084562    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:23:54.185306 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:23:54.185268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:00.841742 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:00.841706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:01.051405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:01.051371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:01.429330 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:01.429294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:01.493313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:01.493280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:02.667936 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.667878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:02.670914 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.670885    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r7fz7"]
Sep 11 18:24:02.673165 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.673143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:02.675722 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.675696    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zr6cv\""
Sep 11 18:24:02.675836 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.675696    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:24:02.676168 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.676150    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 18:24:02.676235 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.676154    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:24:02.683773 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.683746    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r7fz7"]
Sep 11 18:24:02.765270 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.765230    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r7fz7"]
Sep 11 18:24:02.773270 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.773233    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:02.811345 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.811305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fbzh\" (UniqueName: \"kubernetes.io/projected/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-kube-api-access-4fbzh\") pod \"limitador-limitador-8588749977-r7fz7\" (UID: \"b44b1f63-fb6f-4aab-b2d7-f94f159e7129\") " pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:02.811520 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.811380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-config-file\") pod \"limitador-limitador-8588749977-r7fz7\" (UID: \"b44b1f63-fb6f-4aab-b2d7-f94f159e7129\") " pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:02.845625 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.845594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:02.912718 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.912683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-config-file\") pod \"limitador-limitador-8588749977-r7fz7\" (UID: \"b44b1f63-fb6f-4aab-b2d7-f94f159e7129\") " pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:02.912880 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.912750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbzh\" (UniqueName: \"kubernetes.io/projected/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-kube-api-access-4fbzh\") pod \"limitador-limitador-8588749977-r7fz7\" (UID: \"b44b1f63-fb6f-4aab-b2d7-f94f159e7129\") " pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:02.913366 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.913346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-config-file\") pod \"limitador-limitador-8588749977-r7fz7\" (UID: \"b44b1f63-fb6f-4aab-b2d7-f94f159e7129\") " pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:02.922342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.922271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbzh\" (UniqueName: \"kubernetes.io/projected/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-kube-api-access-4fbzh\") pod \"limitador-limitador-8588749977-r7fz7\" (UID: \"b44b1f63-fb6f-4aab-b2d7-f94f159e7129\") " pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:02.983626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:02.983584    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:03.252915 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:03.252889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r7fz7"]
Sep 11 18:24:03.414783 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:03.414741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7" event={"ID":"b44b1f63-fb6f-4aab-b2d7-f94f159e7129","Type":"ContainerStarted","Data":"24e21f9075b117912b8255decb33ff647347108203664752da5d503b71723e50"}
Sep 11 18:24:03.542812 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:03.542711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:03.645581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:03.645544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:06.431783 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:06.431742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7" event={"ID":"b44b1f63-fb6f-4aab-b2d7-f94f159e7129","Type":"ContainerStarted","Data":"fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da"}
Sep 11 18:24:06.432185 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:06.431846    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:06.448333 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:06.448275    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7" podStartSLOduration=2.006742239 podStartE2EDuration="4.448257801s" totalImagesPullingTime="2.441515562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:06.447506014 +0000 UTC m=+527.544099658" watchObservedRunningTime="2026-09-11 18:24:06.448257801 +0000 UTC m=+527.544851432"
Sep 11 18:24:16.129426 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:16.128375    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r7fz7"]
Sep 11 18:24:16.129426 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:16.128710    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7" podUID="b44b1f63-fb6f-4aab-b2d7-f94f159e7129" containerName="limitador" containerID="cri-o://fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da" gracePeriod=30
Sep 11 18:24:16.132444 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:16.132252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:16.665947 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:16.665913    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:16.830119 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:16.830005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fbzh\" (UniqueName: \"kubernetes.io/projected/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-kube-api-access-4fbzh\") pod \"b44b1f63-fb6f-4aab-b2d7-f94f159e7129\" (UID: \"b44b1f63-fb6f-4aab-b2d7-f94f159e7129\") "
Sep 11 18:24:16.830119 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:16.830052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-config-file\") pod \"b44b1f63-fb6f-4aab-b2d7-f94f159e7129\" (UID: \"b44b1f63-fb6f-4aab-b2d7-f94f159e7129\") "
Sep 11 18:24:16.830506 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:16.830478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-config-file" (OuterVolumeSpecName: "config-file") pod "b44b1f63-fb6f-4aab-b2d7-f94f159e7129" (UID: "b44b1f63-fb6f-4aab-b2d7-f94f159e7129"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:24:16.832357 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:16.832326    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-kube-api-access-4fbzh" (OuterVolumeSpecName: "kube-api-access-4fbzh") pod "b44b1f63-fb6f-4aab-b2d7-f94f159e7129" (UID: "b44b1f63-fb6f-4aab-b2d7-f94f159e7129"). InnerVolumeSpecName "kube-api-access-4fbzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:16.931551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:16.931512    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fbzh\" (UniqueName: \"kubernetes.io/projected/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-kube-api-access-4fbzh\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:24:16.931551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:16.931544    2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b44b1f63-fb6f-4aab-b2d7-f94f159e7129-config-file\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:24:17.475838 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:17.475799    2580 generic.go:356] "Generic (PLEG): container finished" podID="b44b1f63-fb6f-4aab-b2d7-f94f159e7129" containerID="fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da" exitCode=0
Sep 11 18:24:17.476352 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:17.475847    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7"
Sep 11 18:24:17.476352 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:17.475891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7" event={"ID":"b44b1f63-fb6f-4aab-b2d7-f94f159e7129","Type":"ContainerDied","Data":"fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da"}
Sep 11 18:24:17.476352 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:17.475942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r7fz7" event={"ID":"b44b1f63-fb6f-4aab-b2d7-f94f159e7129","Type":"ContainerDied","Data":"24e21f9075b117912b8255decb33ff647347108203664752da5d503b71723e50"}
Sep 11 18:24:17.476352 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:17.475962    2580 scope.go:117] "RemoveContainer" containerID="fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da"
Sep 11 18:24:17.484623 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:17.484600    2580 scope.go:117] "RemoveContainer" containerID="fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da"
Sep 11 18:24:17.484951 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:24:17.484927    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da\": container with ID starting with fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da not found: ID does not exist" containerID="fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da"
Sep 11 18:24:17.484998 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:17.484959    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da"} err="failed to get container status \"fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da\": rpc error: code = NotFound desc = could not find container \"fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da\": container with ID starting with fbc600e89c443d0db5c4e3ddc7acc4fed29581d5d7ed90e9d11d457dcfdb93da not found: ID does not exist"
Sep 11 18:24:17.493950 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:17.493915    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r7fz7"]
Sep 11 18:24:17.500120 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:17.500094    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r7fz7"]
Sep 11 18:24:18.768840 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:18.768801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:18.871447 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:18.871407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:19.181086 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.181051    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"]
Sep 11 18:24:19.181363 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.181349    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b1f63-fb6f-4aab-b2d7-f94f159e7129" containerName="limitador"
Sep 11 18:24:19.181363 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.181364    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b1f63-fb6f-4aab-b2d7-f94f159e7129" containerName="limitador"
Sep 11 18:24:19.181460 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.181422    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44b1f63-fb6f-4aab-b2d7-f94f159e7129" containerName="limitador"
Sep 11 18:24:19.185565 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.185539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"
Sep 11 18:24:19.187652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.187629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 18:24:19.187835 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.187818    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nc4tw\""
Sep 11 18:24:19.187909 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.187820    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:24:19.188397 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.188381    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:24:19.192796 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.192770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"]
Sep 11 18:24:19.213810 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.213773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:19.249990 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.249956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfq8n\" (UniqueName: \"kubernetes.io/projected/e866b959-13df-4f37-a273-749b4747d0b4-kube-api-access-bfq8n\") pod \"postgres-69fb4594cf-92lgb\" (UID: \"e866b959-13df-4f37-a273-749b4747d0b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"
Sep 11 18:24:19.249990 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.249991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e866b959-13df-4f37-a273-749b4747d0b4-data\") pod \"postgres-69fb4594cf-92lgb\" (UID: \"e866b959-13df-4f37-a273-749b4747d0b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"
Sep 11 18:24:19.350390 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.350358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfq8n\" (UniqueName: \"kubernetes.io/projected/e866b959-13df-4f37-a273-749b4747d0b4-kube-api-access-bfq8n\") pod \"postgres-69fb4594cf-92lgb\" (UID: \"e866b959-13df-4f37-a273-749b4747d0b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"
Sep 11 18:24:19.350390 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.350395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e866b959-13df-4f37-a273-749b4747d0b4-data\") pod \"postgres-69fb4594cf-92lgb\" (UID: \"e866b959-13df-4f37-a273-749b4747d0b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"
Sep 11 18:24:19.350839 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.350818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e866b959-13df-4f37-a273-749b4747d0b4-data\") pod \"postgres-69fb4594cf-92lgb\" (UID: \"e866b959-13df-4f37-a273-749b4747d0b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"
Sep 11 18:24:19.360893 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.360864    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:24:19.371833 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.371809    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:24:19.382260 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.382226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfq8n\" (UniqueName: \"kubernetes.io/projected/e866b959-13df-4f37-a273-749b4747d0b4-kube-api-access-bfq8n\") pod \"postgres-69fb4594cf-92lgb\" (UID: \"e866b959-13df-4f37-a273-749b4747d0b4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"
Sep 11 18:24:19.397553 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.397513    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44b1f63-fb6f-4aab-b2d7-f94f159e7129" path="/var/lib/kubelet/pods/b44b1f63-fb6f-4aab-b2d7-f94f159e7129/volumes"
Sep 11 18:24:19.500704 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.500604    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nc4tw\""
Sep 11 18:24:19.508572 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.508541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"
Sep 11 18:24:19.779906 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:19.779814    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"]
Sep 11 18:24:19.782865 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:24:19.782826    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode866b959_13df_4f37_a273_749b4747d0b4.slice/crio-cff0715699961515b526e9401d73b4c09baf6c99702611adb7dd8388859bc45c WatchSource:0}: Error finding container cff0715699961515b526e9401d73b4c09baf6c99702611adb7dd8388859bc45c: Status 404 returned error can't find the container with id cff0715699961515b526e9401d73b4c09baf6c99702611adb7dd8388859bc45c
Sep 11 18:24:20.489998 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:20.489963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb" event={"ID":"e866b959-13df-4f37-a273-749b4747d0b4","Type":"ContainerStarted","Data":"cff0715699961515b526e9401d73b4c09baf6c99702611adb7dd8388859bc45c"}
Sep 11 18:24:24.638546 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:24.638522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 18:24:25.512739 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:25.512701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb" event={"ID":"e866b959-13df-4f37-a273-749b4747d0b4","Type":"ContainerStarted","Data":"4e87d4fb3b9c1708edc8ee4b3fbb958f24bde0de838c1661f989c8dce41e0d98"}
Sep 11 18:24:25.512916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:25.512774    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"
Sep 11 18:24:31.545247 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:31.545211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb"
Sep 11 18:24:31.562048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:31.561992    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-92lgb" podStartSLOduration=7.709948045 podStartE2EDuration="12.561975621s" totalImagesPullingTime="4.852027576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:25.539614265 +0000 UTC m=+546.636207896" watchObservedRunningTime="2026-09-11 18:24:31.561975621 +0000 UTC m=+552.658569251"
Sep 11 18:24:31.860468 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:31.860379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:32.058408 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:32.058314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:38.534561 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.534522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:38.573859 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.573824    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-566647dbf9-w25gp"]
Sep 11 18:24:38.595389 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.595352    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-566647dbf9-w25gp"]
Sep 11 18:24:38.595572 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.595482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.597886 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.597863    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 18:24:38.598642 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.598619    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jg26l\""
Sep 11 18:24:38.598794 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.598764    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:24:38.598962 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.598840    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 18:24:38.617536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.617505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:38.677835 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.677804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:38.713145 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.713105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-metrics-certs\") pod \"maas-controller-566647dbf9-w25gp\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") " pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.713361 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.713162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptszr\" (UniqueName: \"kubernetes.io/projected/1f500f70-a95e-44fb-aaec-1a22f9b6145c-kube-api-access-ptszr\") pod \"maas-controller-566647dbf9-w25gp\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") " pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.713361 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.713255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-webhook-cert\") pod \"maas-controller-566647dbf9-w25gp\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") " pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.753313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.753272    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dd795d9-bqnzv"]
Sep 11 18:24:38.755548 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.755529    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:38.766868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.766831    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd795d9-bqnzv"]
Sep 11 18:24:38.772979 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.772942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:38.814489 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.814382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptszr\" (UniqueName: \"kubernetes.io/projected/1f500f70-a95e-44fb-aaec-1a22f9b6145c-kube-api-access-ptszr\") pod \"maas-controller-566647dbf9-w25gp\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") " pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.814489 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.814433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxl8p\" (UniqueName: \"kubernetes.io/projected/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-kube-api-access-wxl8p\") pod \"maas-controller-6dd795d9-bqnzv\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") " pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:38.814489 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.814455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-webhook-cert\") pod \"maas-controller-566647dbf9-w25gp\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") " pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.814489 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.814483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-webhook-cert\") pod \"maas-controller-6dd795d9-bqnzv\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") " pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:38.814489 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.814523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-metrics-certs\") pod \"maas-controller-566647dbf9-w25gp\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") " pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.814879 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.814549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-metrics-certs\") pod \"maas-controller-6dd795d9-bqnzv\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") " pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:38.817004 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.816979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-webhook-cert\") pod \"maas-controller-566647dbf9-w25gp\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") " pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.817093 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.817020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-metrics-certs\") pod \"maas-controller-566647dbf9-w25gp\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") " pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.824070 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.824034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptszr\" (UniqueName: \"kubernetes.io/projected/1f500f70-a95e-44fb-aaec-1a22f9b6145c-kube-api-access-ptszr\") pod \"maas-controller-566647dbf9-w25gp\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") " pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.906921 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.906868    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:38.915177 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.915142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxl8p\" (UniqueName: \"kubernetes.io/projected/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-kube-api-access-wxl8p\") pod \"maas-controller-6dd795d9-bqnzv\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") " pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:38.915313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.915192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-webhook-cert\") pod \"maas-controller-6dd795d9-bqnzv\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") " pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:38.915313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.915276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-metrics-certs\") pod \"maas-controller-6dd795d9-bqnzv\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") " pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:38.917782 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.917758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-metrics-certs\") pod \"maas-controller-6dd795d9-bqnzv\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") " pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:38.917881 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.917784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-webhook-cert\") pod \"maas-controller-6dd795d9-bqnzv\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") " pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:38.924405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:38.924374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxl8p\" (UniqueName: \"kubernetes.io/projected/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-kube-api-access-wxl8p\") pod \"maas-controller-6dd795d9-bqnzv\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") " pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:39.065888 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:39.065790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:39.180738 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:39.180712    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-566647dbf9-w25gp"]
Sep 11 18:24:39.183095 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:24:39.183063    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f500f70_a95e_44fb_aaec_1a22f9b6145c.slice/crio-0af5b92eaefe0794f51c7c575eb2033d028b5c1200f670e7c98eceba1c4343d1 WatchSource:0}: Error finding container 0af5b92eaefe0794f51c7c575eb2033d028b5c1200f670e7c98eceba1c4343d1: Status 404 returned error can't find the container with id 0af5b92eaefe0794f51c7c575eb2033d028b5c1200f670e7c98eceba1c4343d1
Sep 11 18:24:39.337938 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:39.337887    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd795d9-bqnzv"]
Sep 11 18:24:39.339977 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:24:39.339955    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4eedc1_dda7_4d8c_bc45_156bfcbc2b5f.slice/crio-85c5d165cb617a13f3e57676173df7c7118009bcddc804c48f9df26d3b3d03c1 WatchSource:0}: Error finding container 85c5d165cb617a13f3e57676173df7c7118009bcddc804c48f9df26d3b3d03c1: Status 404 returned error can't find the container with id 85c5d165cb617a13f3e57676173df7c7118009bcddc804c48f9df26d3b3d03c1
Sep 11 18:24:39.567795 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:39.567759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566647dbf9-w25gp" event={"ID":"1f500f70-a95e-44fb-aaec-1a22f9b6145c","Type":"ContainerStarted","Data":"0af5b92eaefe0794f51c7c575eb2033d028b5c1200f670e7c98eceba1c4343d1"}
Sep 11 18:24:39.568922 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:39.568894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd795d9-bqnzv" event={"ID":"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f","Type":"ContainerStarted","Data":"85c5d165cb617a13f3e57676173df7c7118009bcddc804c48f9df26d3b3d03c1"}
Sep 11 18:24:42.029023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:42.028981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:42.128686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:42.128649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:24:42.590608 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:42.590516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd795d9-bqnzv" event={"ID":"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f","Type":"ContainerStarted","Data":"2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5"}
Sep 11 18:24:42.590796 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:42.590625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:42.592188 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:42.592160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566647dbf9-w25gp" event={"ID":"1f500f70-a95e-44fb-aaec-1a22f9b6145c","Type":"ContainerStarted","Data":"18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42"}
Sep 11 18:24:42.592333 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:42.592309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:42.608605 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:42.608550    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6dd795d9-bqnzv" podStartSLOduration=2.082545088 podStartE2EDuration="4.608532984s" totalImagesPullingTime="2.525987896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:42.60775493 +0000 UTC m=+563.704348572" watchObservedRunningTime="2026-09-11 18:24:42.608532984 +0000 UTC m=+563.705126681"
Sep 11 18:24:42.628249 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:42.628165    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-566647dbf9-w25gp" podStartSLOduration=1.954907291 podStartE2EDuration="4.628147059s" totalImagesPullingTime="2.673239768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:42.626920353 +0000 UTC m=+563.723513985" watchObservedRunningTime="2026-09-11 18:24:42.628147059 +0000 UTC m=+563.724740690"
Sep 11 18:24:53.598463 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:53.598431    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:53.598954 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:53.598514    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:24:53.658985 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:53.658946    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-566647dbf9-w25gp"]
Sep 11 18:24:53.659180 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:53.659162    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-566647dbf9-w25gp" podUID="1f500f70-a95e-44fb-aaec-1a22f9b6145c" containerName="manager" containerID="cri-o://18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42" gracePeriod=30
Sep 11 18:24:53.907058 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:53.907021    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:54.048309 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.048262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-webhook-cert\") pod \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") "
Sep 11 18:24:54.048505 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.048330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptszr\" (UniqueName: \"kubernetes.io/projected/1f500f70-a95e-44fb-aaec-1a22f9b6145c-kube-api-access-ptszr\") pod \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") "
Sep 11 18:24:54.048505 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.048372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-metrics-certs\") pod \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\" (UID: \"1f500f70-a95e-44fb-aaec-1a22f9b6145c\") "
Sep 11 18:24:54.050723 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.050681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f500f70-a95e-44fb-aaec-1a22f9b6145c-kube-api-access-ptszr" (OuterVolumeSpecName: "kube-api-access-ptszr") pod "1f500f70-a95e-44fb-aaec-1a22f9b6145c" (UID: "1f500f70-a95e-44fb-aaec-1a22f9b6145c"). InnerVolumeSpecName "kube-api-access-ptszr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:54.050723 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.050705    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f500f70-a95e-44fb-aaec-1a22f9b6145c" (UID: "1f500f70-a95e-44fb-aaec-1a22f9b6145c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:24:54.050723 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.050691    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1f500f70-a95e-44fb-aaec-1a22f9b6145c" (UID: "1f500f70-a95e-44fb-aaec-1a22f9b6145c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:24:54.149713 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.149618    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:24:54.149713 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.149650    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f500f70-a95e-44fb-aaec-1a22f9b6145c-webhook-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:24:54.149713 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.149664    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptszr\" (UniqueName: \"kubernetes.io/projected/1f500f70-a95e-44fb-aaec-1a22f9b6145c-kube-api-access-ptszr\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:24:54.644588 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.644556    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f500f70-a95e-44fb-aaec-1a22f9b6145c" containerID="18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42" exitCode=0
Sep 11 18:24:54.645048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.644601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566647dbf9-w25gp" event={"ID":"1f500f70-a95e-44fb-aaec-1a22f9b6145c","Type":"ContainerDied","Data":"18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42"}
Sep 11 18:24:54.645048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.644639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566647dbf9-w25gp" event={"ID":"1f500f70-a95e-44fb-aaec-1a22f9b6145c","Type":"ContainerDied","Data":"0af5b92eaefe0794f51c7c575eb2033d028b5c1200f670e7c98eceba1c4343d1"}
Sep 11 18:24:54.645048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.644659    2580 scope.go:117] "RemoveContainer" containerID="18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42"
Sep 11 18:24:54.645048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.644602    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566647dbf9-w25gp"
Sep 11 18:24:54.659287 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.659263    2580 scope.go:117] "RemoveContainer" containerID="18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42"
Sep 11 18:24:54.659638 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:24:54.659617    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42\": container with ID starting with 18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42 not found: ID does not exist" containerID="18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42"
Sep 11 18:24:54.659682 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.659648    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42"} err="failed to get container status \"18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42\": rpc error: code = NotFound desc = could not find container \"18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42\": container with ID starting with 18393da2ee0feb8c60b35071f9d51b32862e0d6330d21657fdc069b19b102a42 not found: ID does not exist"
Sep 11 18:24:54.670864 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.670833    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-566647dbf9-w25gp"]
Sep 11 18:24:54.675486 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:54.675450    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-566647dbf9-w25gp"]
Sep 11 18:24:55.394999 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:24:55.394962    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f500f70-a95e-44fb-aaec-1a22f9b6145c" path="/var/lib/kubelet/pods/1f500f70-a95e-44fb-aaec-1a22f9b6145c/volumes"
Sep 11 18:25:13.552037 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:13.551974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:13.654467 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:13.654428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:15.145105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.145062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:15.251457 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.251423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:15.284818 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.284780    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"]
Sep 11 18:25:15.285141 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.285126    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f500f70-a95e-44fb-aaec-1a22f9b6145c" containerName="manager"
Sep 11 18:25:15.285193 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.285142    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f500f70-a95e-44fb-aaec-1a22f9b6145c" containerName="manager"
Sep 11 18:25:15.285193 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.285190    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f500f70-a95e-44fb-aaec-1a22f9b6145c" containerName="manager"
Sep 11 18:25:15.303390 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.303352    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"]
Sep 11 18:25:15.303562 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.303495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.305981 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.305956    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:25:15.306755 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.306729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c9xd9\""
Sep 11 18:25:15.306870 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.306776    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 18:25:15.373498 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.373459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:15.415126 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.415035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-metrics-certs\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.415126 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.415112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-maas-api-tls\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.415332 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.415150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bj6k\" (UniqueName: \"kubernetes.io/projected/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-kube-api-access-8bj6k\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.415332 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.415171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-openshift-service-ca\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.516466 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.516423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-maas-api-tls\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.516662 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.516487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bj6k\" (UniqueName: \"kubernetes.io/projected/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-kube-api-access-8bj6k\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.516662 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.516517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-openshift-service-ca\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.516662 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.516561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-metrics-certs\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.517226 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.517179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-openshift-service-ca\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.519471 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.519432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-metrics-certs\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.519778 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.519758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-maas-api-tls\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.527649 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.527619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bj6k\" (UniqueName: \"kubernetes.io/projected/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-kube-api-access-8bj6k\") pod \"maas-api-6b5bd7959c-5spcc\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.614595 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.614561    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:15.706304 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.706229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:15.802698 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.802645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:15.895040 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:15.895011    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"]
Sep 11 18:25:15.896885 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:25:15.896853    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d9a2f8_9fec_440e_8f95_9c5cf5dbc6ae.slice/crio-87f0b6e207c0f0dde6552cdd73514211605d9d168601b2bd85f8be1d5ccabbed WatchSource:0}: Error finding container 87f0b6e207c0f0dde6552cdd73514211605d9d168601b2bd85f8be1d5ccabbed: Status 404 returned error can't find the container with id 87f0b6e207c0f0dde6552cdd73514211605d9d168601b2bd85f8be1d5ccabbed
Sep 11 18:25:16.743845 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:16.743805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc" event={"ID":"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae","Type":"ContainerStarted","Data":"87f0b6e207c0f0dde6552cdd73514211605d9d168601b2bd85f8be1d5ccabbed"}
Sep 11 18:25:18.755268 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:18.755231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc" event={"ID":"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae","Type":"ContainerStarted","Data":"d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b"}
Sep 11 18:25:18.755717 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:18.755356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:18.774284 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:18.774222    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc" podStartSLOduration=1.915768675 podStartE2EDuration="3.774188789s" totalImagesPullingTime="1.858420114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:18.772494567 +0000 UTC m=+599.869088213" watchObservedRunningTime="2026-09-11 18:25:18.774188789 +0000 UTC m=+599.870782420"
Sep 11 18:25:24.765038 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:24.765006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:25:25.736998 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:25.736956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:25.839820 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:25.839769    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:30.093114 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:30.093067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:41.538108 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.538014    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"]
Sep 11 18:25:41.541247 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.541223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.544303 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.544271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 18:25:41.544303 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.544296    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 18:25:41.544509 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.544299    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 18:25:41.544628 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.544614    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zv74c\""
Sep 11 18:25:41.552074 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.552047    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"]
Sep 11 18:25:41.573560 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.573524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:41.638041 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.637994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.638041 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.638042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.638311 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.638119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.638311 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.638193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.638311 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.638278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c9zh\" (UniqueName: \"kubernetes.io/projected/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-kube-api-access-5c9zh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.638311 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.638300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.739509 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.739464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.739509 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.739514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.739767 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.739556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c9zh\" (UniqueName: \"kubernetes.io/projected/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-kube-api-access-5c9zh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.739767 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.739584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.739767 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.739717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.739767 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.739757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.740034 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.739979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.740034 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.740003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.740101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.740064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.741764 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.741738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.742185 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.742167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.748376 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.748341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c9zh\" (UniqueName: \"kubernetes.io/projected/a439c393-bf3a-4de7-ad06-00ee2caf5b9f-kube-api-access-5c9zh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9hw2w\" (UID: \"a439c393-bf3a-4de7-ad06-00ee2caf5b9f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:41.853026 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:41.852944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:42.146550 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:42.146519    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"]
Sep 11 18:25:42.148935 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:25:42.148908    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda439c393_bf3a_4de7_ad06_00ee2caf5b9f.slice/crio-7d12c4ac93dc69c0993347ac756e4bf6f05e6e7eed3c2b181e8047ce73d0a3d3 WatchSource:0}: Error finding container 7d12c4ac93dc69c0993347ac756e4bf6f05e6e7eed3c2b181e8047ce73d0a3d3: Status 404 returned error can't find the container with id 7d12c4ac93dc69c0993347ac756e4bf6f05e6e7eed3c2b181e8047ce73d0a3d3
Sep 11 18:25:42.860823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:42.860783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w" event={"ID":"a439c393-bf3a-4de7-ad06-00ee2caf5b9f","Type":"ContainerStarted","Data":"7d12c4ac93dc69c0993347ac756e4bf6f05e6e7eed3c2b181e8047ce73d0a3d3"}
Sep 11 18:25:43.867698 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:43.867652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w" event={"ID":"a439c393-bf3a-4de7-ad06-00ee2caf5b9f","Type":"ContainerStarted","Data":"30dca20df0431f624b827e1a8f445e16d4721f936b950a4823d63689e8173103"}
Sep 11 18:25:43.885629 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:43.885567    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w" podStartSLOduration=1.8986713640000001 podStartE2EDuration="2.885541993s" totalImagesPullingTime="986.870629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:43.883749026 +0000 UTC m=+624.980342657" watchObservedRunningTime="2026-09-11 18:25:43.885541993 +0000 UTC m=+624.982135623"
Sep 11 18:25:46.179080 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:46.179026    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:25:51.853188 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:51.853137    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:51.853804 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:51.853245    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:51.865884 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:51.865847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:51.912683 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:51.912646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9hw2w"
Sep 11 18:25:57.373620 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:25:57.373580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:26:04.618143 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:04.618103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:26:05.143587 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.143547    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"]
Sep 11 18:26:05.145972 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.145946    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.148633 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.148610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 18:26:05.150661 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.150639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.150780 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.150694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.150780 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.150740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.150872 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.150800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae642ae-41a7-431f-8a78-b487d15d3fae-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.150872 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.150835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.150951 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.150882    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdq4r\" (UniqueName: \"kubernetes.io/projected/5ae642ae-41a7-431f-8a78-b487d15d3fae-kube-api-access-hdq4r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.156519 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.156492    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"]
Sep 11 18:26:05.251448 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.251402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdq4r\" (UniqueName: \"kubernetes.io/projected/5ae642ae-41a7-431f-8a78-b487d15d3fae-kube-api-access-hdq4r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.251632 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.251461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.251632 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.251487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.251632 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.251509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.251632 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.251536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae642ae-41a7-431f-8a78-b487d15d3fae-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.251632 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.251561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.252012 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.251987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.252095 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.251998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.252095 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.252078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.253857 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.253836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae642ae-41a7-431f-8a78-b487d15d3fae-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.254183 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.254164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae642ae-41a7-431f-8a78-b487d15d3fae-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.260035 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.260003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdq4r\" (UniqueName: \"kubernetes.io/projected/5ae642ae-41a7-431f-8a78-b487d15d3fae-kube-api-access-hdq4r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg\" (UID: \"5ae642ae-41a7-431f-8a78-b487d15d3fae\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.456850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.456747    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:05.728811 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.728728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"]
Sep 11 18:26:05.732226 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:26:05.732169    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae642ae_41a7_431f_8a78_b487d15d3fae.slice/crio-9902c6df38f9015a24b089a9d85a08831b956dd8bc30e68fd600fd0bd333b88e WatchSource:0}: Error finding container 9902c6df38f9015a24b089a9d85a08831b956dd8bc30e68fd600fd0bd333b88e: Status 404 returned error can't find the container with id 9902c6df38f9015a24b089a9d85a08831b956dd8bc30e68fd600fd0bd333b88e
Sep 11 18:26:05.733635 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.733612    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:26:05.963653 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.963615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg" event={"ID":"5ae642ae-41a7-431f-8a78-b487d15d3fae","Type":"ContainerStarted","Data":"d98524fbf8e2067e115f8aea878e93a85afbcee876b6d944d472ccce65307c32"}
Sep 11 18:26:05.963840 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.963669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg" event={"ID":"5ae642ae-41a7-431f-8a78-b487d15d3fae","Type":"ContainerStarted","Data":"9902c6df38f9015a24b089a9d85a08831b956dd8bc30e68fd600fd0bd333b88e"}
Sep 11 18:26:05.981449 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:05.981350    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg" podStartSLOduration=0.810444389 podStartE2EDuration="981.328308ms" totalImagesPullingTime="170.883919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:05.979538269 +0000 UTC m=+647.076131914" watchObservedRunningTime="2026-09-11 18:26:05.981328308 +0000 UTC m=+647.077921938"
Sep 11 18:26:06.032579 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:06.032541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:26:09.228138 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:09.228099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:26:09.838515 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:09.838480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:26:12.054100 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:12.054056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:26:15.457671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:15.457623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:15.458085 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:15.457707    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:15.470256 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:15.470228    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:16.018426 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:16.018396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg"
Sep 11 18:26:21.251563 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:26:21.251523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:15.330874 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:15.330784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:15.431300 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:15.431255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:16.471898 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:16.471862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:16.568098 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:16.568063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:29.492555 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:29.492519    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:31.735468 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:31.735420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:31.835785 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:31.835728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:31.943972 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:31.943924    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:32.133446 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.133405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:32.226758 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.226713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:32.227464 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.227441    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"]
Sep 11 18:27:32.230233 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.230186    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.232664 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.232642    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-724327-openshift-default-dockercfg-9nzt7\""
Sep 11 18:27:32.243108 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.243074    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"]
Sep 11 18:27:32.296767 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.296729    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:32.380903 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.380864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-data\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.381095 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.380914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istiod-ca-cert\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.381095 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.380944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4jgp\" (UniqueName: \"kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-kube-api-access-q4jgp\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.381095 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.381028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-credential-socket\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.381289 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.381099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-socket\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.381289 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.381126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-certs\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.381289 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.381189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-envoy\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.381401 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.381282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-podinfo\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.381401 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.381361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-token\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.482585 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.482495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-envoy\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.482585 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.482545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-podinfo\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.482809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.482639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-token\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.482809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.482680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-data\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.482809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.482738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istiod-ca-cert\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.482809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.482785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4jgp\" (UniqueName: \"kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-kube-api-access-q4jgp\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.483024 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.482809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-credential-socket\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.483024 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.482851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-socket\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.483024 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.482876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-certs\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.483194 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.483078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-data\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.483194 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.483176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-certs\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.483391 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.483344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-credential-socket\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.483391 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.483375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-socket\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.483516 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.483498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istiod-ca-cert\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.484850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.484830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-envoy\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.485089 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.485068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-podinfo\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.491100 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.491069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-token\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.491271 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.491211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4jgp\" (UniqueName: \"kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-kube-api-access-q4jgp\") pod \"e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") " pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.541906 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.541859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:32.600915 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.600881    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:32.696353 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.696316    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:32.817478 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:32.817452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"]
Sep 11 18:27:32.819533 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:27:32.819505    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49cbaa6_1faa_45ed_8577_f474aeb73a9b.slice/crio-c4917a38bfcccf44e47a66a3e78778106bb159c06a181482beccc7fd30752513 WatchSource:0}: Error finding container c4917a38bfcccf44e47a66a3e78778106bb159c06a181482beccc7fd30752513: Status 404 returned error can't find the container with id c4917a38bfcccf44e47a66a3e78778106bb159c06a181482beccc7fd30752513
Sep 11 18:27:33.126933 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:33.126895    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:33.225503 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:33.225459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:33.340960 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:33.340919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" event={"ID":"d49cbaa6-1faa-45ed-8577-f474aeb73a9b","Type":"ContainerStarted","Data":"c4917a38bfcccf44e47a66a3e78778106bb159c06a181482beccc7fd30752513"}
Sep 11 18:27:33.485145 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:33.485051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:33.587780 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:33.587740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:34.929047 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:34.929001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:35.031738 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:35.031702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:35.137012 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:35.136974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:35.725025 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:35.724981    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:27:35.725155 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:35.725074    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:27:35.725155 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:35.725117    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:27:36.211637 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.211599    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"]
Sep 11 18:27:36.218629 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.215261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.218629 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.218369    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-efa8b9-serving-cert\""
Sep 11 18:27:36.228229 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.228186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"]
Sep 11 18:27:36.245646 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.245593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:36.321586 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.321536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3652753b-1774-45b0-b34b-6b06eb31a678-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.321586 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.321587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd8np\" (UniqueName: \"kubernetes.io/projected/3652753b-1774-45b0-b34b-6b06eb31a678-kube-api-access-rd8np\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.321805 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.321649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-maas-api-tls\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.321805 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.321748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-metrics-certs\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.345658 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.345621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:36.359546 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.359503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" event={"ID":"d49cbaa6-1faa-45ed-8577-f474aeb73a9b","Type":"ContainerStarted","Data":"23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e"}
Sep 11 18:27:36.390327 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.390257    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" podStartSLOduration=1.487065871 podStartE2EDuration="4.390237056s" totalImagesPullingTime="2.903171185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:36.387902369 +0000 UTC m=+737.484496001" watchObservedRunningTime="2026-09-11 18:27:36.390237056 +0000 UTC m=+737.486830688"
Sep 11 18:27:36.423261 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.423184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3652753b-1774-45b0-b34b-6b06eb31a678-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.423261 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.423259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8np\" (UniqueName: \"kubernetes.io/projected/3652753b-1774-45b0-b34b-6b06eb31a678-kube-api-access-rd8np\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.423584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.423282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-maas-api-tls\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.423584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.423371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-metrics-certs\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.424030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.423995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3652753b-1774-45b0-b34b-6b06eb31a678-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.425938 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.425904    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-metrics-certs\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.426050 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.425954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-maas-api-tls\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.431923 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.431891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8np\" (UniqueName: \"kubernetes.io/projected/3652753b-1774-45b0-b34b-6b06eb31a678-kube-api-access-rd8np\") pod \"maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.445784 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.445746    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:36.531615 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.531515    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:36.542738 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.542688    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:36.549463 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.549426    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:36.772849 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.772801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:36.810685 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.810648    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"]
Sep 11 18:27:36.813381 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:27:36.813332    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3652753b_1774_45b0_b34b_6b06eb31a678.slice/crio-37b929b3c0155b717b3744f7d9c5cb317eb60ddcb1d052803d36e506d77c5a78 WatchSource:0}: Error finding container 37b929b3c0155b717b3744f7d9c5cb317eb60ddcb1d052803d36e506d77c5a78: Status 404 returned error can't find the container with id 37b929b3c0155b717b3744f7d9c5cb317eb60ddcb1d052803d36e506d77c5a78
Sep 11 18:27:36.873717 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:36.873677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:37.318643 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.318601    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"]
Sep 11 18:27:37.322102 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.322078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.324410 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.324378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7d7509-serving-cert\""
Sep 11 18:27:37.331382 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.331028    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"]
Sep 11 18:27:37.360758 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.360717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:37.366729 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.366683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97" event={"ID":"3652753b-1774-45b0-b34b-6b06eb31a678","Type":"ContainerStarted","Data":"37b929b3c0155b717b3744f7d9c5cb317eb60ddcb1d052803d36e506d77c5a78"}
Sep 11 18:27:37.366952 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.366935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:37.368009 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.367986    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:27:37.434008 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.433977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.434179 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.434050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.434179 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.434074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-metrics-certs\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.434179 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.434090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjmr5\" (UniqueName: \"kubernetes.io/projected/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-kube-api-access-fjmr5\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.462043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.462001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:37.534900 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.534854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.535218 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.535162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.535379 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.535244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-metrics-certs\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.535379 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.535273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjmr5\" (UniqueName: \"kubernetes.io/projected/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-kube-api-access-fjmr5\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.535710 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.535689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.537703 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.537683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.537793 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.537732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-metrics-certs\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.544451 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.544418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjmr5\" (UniqueName: \"kubernetes.io/projected/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-kube-api-access-fjmr5\") pod \"maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.561498 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.561463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:37.636710 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.636669    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:37.910470 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:37.910387    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"]
Sep 11 18:27:37.914133 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:27:37.914099    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849cdd3b_c6a7_4279_8c1b_2ec478946f0c.slice/crio-af5bbc7f5b038207fc0d36e94f8f0fd507b3e05073a0f74c884742d7cc2dc248 WatchSource:0}: Error finding container af5bbc7f5b038207fc0d36e94f8f0fd507b3e05073a0f74c884742d7cc2dc248: Status 404 returned error can't find the container with id af5bbc7f5b038207fc0d36e94f8f0fd507b3e05073a0f74c884742d7cc2dc248
Sep 11 18:27:38.006824 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.006780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:38.083533 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.083501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:38.372240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.372172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97" event={"ID":"3652753b-1774-45b0-b34b-6b06eb31a678","Type":"ContainerStarted","Data":"00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6"}
Sep 11 18:27:38.372736 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.372270    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:38.373825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.373802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf" event={"ID":"849cdd3b-c6a7-4279-8c1b-2ec478946f0c","Type":"ContainerStarted","Data":"3fbe4d10f0b9717e38ba1624994c404ac20366c6ce74322489283737350ce563"}
Sep 11 18:27:38.373936 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.373848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf" event={"ID":"849cdd3b-c6a7-4279-8c1b-2ec478946f0c","Type":"ContainerStarted","Data":"af5bbc7f5b038207fc0d36e94f8f0fd507b3e05073a0f74c884742d7cc2dc248"}
Sep 11 18:27:38.373982 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.373934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:38.396663 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.396606    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97" podStartSLOduration=1.8081551519999999 podStartE2EDuration="2.396588269s" totalImagesPullingTime="588.433117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:38.395688556 +0000 UTC m=+739.492282191" watchObservedRunningTime="2026-09-11 18:27:38.396588269 +0000 UTC m=+739.493181899"
Sep 11 18:27:38.425151 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.425082    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf" podStartSLOduration=1.132931564 podStartE2EDuration="1.425060026s" totalImagesPullingTime="292.128462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:38.423733783 +0000 UTC m=+739.520327426" watchObservedRunningTime="2026-09-11 18:27:38.425060026 +0000 UTC m=+739.521653656"
Sep 11 18:27:38.454816 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.454761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:38.553693 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:38.553588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:39.605379 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:39.605333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:39.774583 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:39.774528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:39.875147 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:39.875107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:40.181494 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:40.181388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:40.281208 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:40.281153    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:40.385495 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:40.385450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:40.473671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:40.473583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:40.548938 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:40.548899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:40.834654 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:40.834540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:40.928930 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:40.928868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:41.061596 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:41.061560    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:41.332584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:41.332543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:42.206388 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.206351    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"]
Sep 11 18:27:42.209897 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.209865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.212176 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.212144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-724327-serving-cert\""
Sep 11 18:27:42.220352 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.220315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"]
Sep 11 18:27:42.239890 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.239840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:42.341595 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.341546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:42.382946 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.382908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e8f0b7e-49db-4650-8152-923be3fd1278-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.383143 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.382959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-maas-api-tls\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.383143 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.382993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqctr\" (UniqueName: \"kubernetes.io/projected/5e8f0b7e-49db-4650-8152-923be3fd1278-kube-api-access-kqctr\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.383143 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.383022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-metrics-certs\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.440557 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.440518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:42.483766 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.483654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e8f0b7e-49db-4650-8152-923be3fd1278-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.483766 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.483712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-maas-api-tls\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.484000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.483848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqctr\" (UniqueName: \"kubernetes.io/projected/5e8f0b7e-49db-4650-8152-923be3fd1278-kube-api-access-kqctr\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.484000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.483923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-metrics-certs\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.484495 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.484471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e8f0b7e-49db-4650-8152-923be3fd1278-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.486328 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.486310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-metrics-certs\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.486494 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.486471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-maas-api-tls\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.492166 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.492133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqctr\" (UniqueName: \"kubernetes.io/projected/5e8f0b7e-49db-4650-8152-923be3fd1278-kube-api-access-kqctr\") pod \"maas-api-e2e-worker-w4-724327-86788c7b6f-c227k\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.522337 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.522286    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:42.732854 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.732814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:42.798019 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.797992    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"]
Sep 11 18:27:42.800054 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:42.800021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:42.800373 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:27:42.800338    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8f0b7e_49db_4650_8152_923be3fd1278.slice/crio-62f55dd595ae1307e77587f275c8458b8a938c36519de517b8d9da33e77a8502 WatchSource:0}: Error finding container 62f55dd595ae1307e77587f275c8458b8a938c36519de517b8d9da33e77a8502: Status 404 returned error can't find the container with id 62f55dd595ae1307e77587f275c8458b8a938c36519de517b8d9da33e77a8502
Sep 11 18:27:43.386498 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:43.386461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:43.405317 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:43.405214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k" event={"ID":"5e8f0b7e-49db-4650-8152-923be3fd1278","Type":"ContainerStarted","Data":"0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458"}
Sep 11 18:27:43.405317 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:43.405285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k" event={"ID":"5e8f0b7e-49db-4650-8152-923be3fd1278","Type":"ContainerStarted","Data":"62f55dd595ae1307e77587f275c8458b8a938c36519de517b8d9da33e77a8502"}
Sep 11 18:27:43.405570 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:43.405415    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:43.424704 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:43.424595    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k" podStartSLOduration=1.054859241 podStartE2EDuration="1.424579899s" totalImagesPullingTime="369.720658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:43.42333577 +0000 UTC m=+744.519929401" watchObservedRunningTime="2026-09-11 18:27:43.424579899 +0000 UTC m=+744.521173529"
Sep 11 18:27:43.469841 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:43.469800    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:43.990226 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:43.990159    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:44.084907 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:44.084871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:44.385234 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:44.385181    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:27:44.385561 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:44.385542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:27:44.741529 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:44.741418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:45.704729 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:45.704682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:46.403269 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.403234    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"]
Sep 11 18:27:46.407617 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.407590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.410093 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.410064    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-7e8ab9-serving-cert\""
Sep 11 18:27:46.415140 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.415107    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"]
Sep 11 18:27:46.446254 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.446192    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:46.500959 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.500927    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:46.520698 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.520657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcd2n\" (UniqueName: \"kubernetes.io/projected/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-kube-api-access-zcd2n\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.521028 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.520998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-openshift-service-ca\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.521147 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.521054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-metrics-certs\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.521250 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.521190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-maas-api-tls\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.621698 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.621659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcd2n\" (UniqueName: \"kubernetes.io/projected/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-kube-api-access-zcd2n\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.621915 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.621734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-openshift-service-ca\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.621915 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.621760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-metrics-certs\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.621915 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.621792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-maas-api-tls\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.621915 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:27:46.621899    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-serving-cert: secret "maas-api-e2e-autores-7e8ab9-serving-cert" not found
Sep 11 18:27:46.622115 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:27:46.621968    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-maas-api-tls podName:bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa nodeName:}" failed. No retries permitted until 2026-09-11 18:27:47.121948086 +0000 UTC m=+748.218541712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-maas-api-tls") pod "maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r" (UID: "bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa") : secret "maas-api-e2e-autores-7e8ab9-serving-cert" not found
Sep 11 18:27:46.622601 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.622568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-openshift-service-ca\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.624422 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.624400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-metrics-certs\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:46.630014 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:46.629980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcd2n\" (UniqueName: \"kubernetes.io/projected/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-kube-api-access-zcd2n\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:47.036926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:47.036891    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:47.126091 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:47.126052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-maas-api-tls\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:47.128673 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:47.128647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-maas-api-tls\") pod \"maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:47.139844 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:47.139813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:47.240534 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:47.240493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:47.322961 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:47.322864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:47.512144 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:47.512102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:47.598184 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:47.598155    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"]
Sep 11 18:27:47.600700 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:27:47.600664    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3c08e5_b3ce_44ba_be9c_594f7ce9c0aa.slice/crio-20ce0ea98ea29f66c223f02bdc77b23ac2f5ac49b947e7713cbbb675169b5768 WatchSource:0}: Error finding container 20ce0ea98ea29f66c223f02bdc77b23ac2f5ac49b947e7713cbbb675169b5768: Status 404 returned error can't find the container with id 20ce0ea98ea29f66c223f02bdc77b23ac2f5ac49b947e7713cbbb675169b5768
Sep 11 18:27:48.320125 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.320035    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"]
Sep 11 18:27:48.324306 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.324271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.326561 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.326529    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8370956b-serving-cert\""
Sep 11 18:27:48.334868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.334833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"]
Sep 11 18:27:48.360718 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.360672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:48.420952 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.420916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:48.443067 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.443026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3aa4c5e-2db5-46c5-910a-3175973c0c81-openshift-service-ca\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.443303 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.443077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-metrics-certs\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.443303 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.443153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-maas-api-tls\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.443430 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.443385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8tnb\" (UniqueName: \"kubernetes.io/projected/e3aa4c5e-2db5-46c5-910a-3175973c0c81-kube-api-access-l8tnb\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.443498 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.443466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r" event={"ID":"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa","Type":"ContainerStarted","Data":"fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062"}
Sep 11 18:27:48.443584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.443568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r" event={"ID":"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa","Type":"ContainerStarted","Data":"20ce0ea98ea29f66c223f02bdc77b23ac2f5ac49b947e7713cbbb675169b5768"}
Sep 11 18:27:48.443649 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.443628    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:48.466263 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.466189    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r" podStartSLOduration=2.014272114 podStartE2EDuration="2.466171059s" totalImagesPullingTime="451.898945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:48.46488702 +0000 UTC m=+749.561480653" watchObservedRunningTime="2026-09-11 18:27:48.466171059 +0000 UTC m=+749.562764690"
Sep 11 18:27:48.544775 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.544735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3aa4c5e-2db5-46c5-910a-3175973c0c81-openshift-service-ca\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.545268 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.544782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-metrics-certs\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.545268 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.544810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-maas-api-tls\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.545268 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:27:48.544908    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-serving-cert: secret "maas-api-e2e-ait-8370956b-serving-cert" not found
Sep 11 18:27:48.545268 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:27:48.544981    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-maas-api-tls podName:e3aa4c5e-2db5-46c5-910a-3175973c0c81 nodeName:}" failed. No retries permitted until 2026-09-11 18:27:49.044959612 +0000 UTC m=+750.141553221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-maas-api-tls") pod "maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv" (UID: "e3aa4c5e-2db5-46c5-910a-3175973c0c81") : secret "maas-api-e2e-ait-8370956b-serving-cert" not found
Sep 11 18:27:48.545268 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.545042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8tnb\" (UniqueName: \"kubernetes.io/projected/e3aa4c5e-2db5-46c5-910a-3175973c0c81-kube-api-access-l8tnb\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.545495 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.545458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3aa4c5e-2db5-46c5-910a-3175973c0c81-openshift-service-ca\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.547437 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.547411    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-metrics-certs\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.554687 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.554627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8tnb\" (UniqueName: \"kubernetes.io/projected/e3aa4c5e-2db5-46c5-910a-3175973c0c81-kube-api-access-l8tnb\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:48.859875 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.859787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:48.959565 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:48.959531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:49.038230 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:49.038178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:49.049324 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:49.049291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-maas-api-tls\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:49.051747 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:49.051719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-maas-api-tls\") pod \"maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:49.237082 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:49.237040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:49.342829 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:49.342787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:49.415091 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:49.415058    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:27:49.519623 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:49.519533    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"]
Sep 11 18:27:49.522697 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:27:49.522658    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3aa4c5e_2db5_46c5_910a_3175973c0c81.slice/crio-0db74ef380eeb76b33e71c07d931197ba26e9296a279bb17114ba14df941428f WatchSource:0}: Error finding container 0db74ef380eeb76b33e71c07d931197ba26e9296a279bb17114ba14df941428f: Status 404 returned error can't find the container with id 0db74ef380eeb76b33e71c07d931197ba26e9296a279bb17114ba14df941428f
Sep 11 18:27:49.542466 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:49.542407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:49.671625 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:49.671591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:49.973860 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:49.973816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:50.075832 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:50.075794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:50.178502 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:50.178465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:50.313971 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:50.313918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:50.449065 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:50.448967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:50.454774 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:50.454734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv" event={"ID":"e3aa4c5e-2db5-46c5-910a-3175973c0c81","Type":"ContainerStarted","Data":"3119bbc00813e43b0240141d94079b80b19e200023da271d46edf8bf51502633"}
Sep 11 18:27:50.454927 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:50.454819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:50.454927 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:50.454844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv" event={"ID":"e3aa4c5e-2db5-46c5-910a-3175973c0c81","Type":"ContainerStarted","Data":"0db74ef380eeb76b33e71c07d931197ba26e9296a279bb17114ba14df941428f"}
Sep 11 18:27:50.471695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:50.471642    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv" podStartSLOduration=2.144874238 podStartE2EDuration="2.471624979s" totalImagesPullingTime="326.750741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:50.470824424 +0000 UTC m=+751.567418068" watchObservedRunningTime="2026-09-11 18:27:50.471624979 +0000 UTC m=+751.568218643"
Sep 11 18:27:50.542514 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:50.542476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:51.250955 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:51.250913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:52.227753 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:52.227713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:52.398241 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:52.398209    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:52.548100 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:52.548007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:52.699580 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:52.699528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:52.799888 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:52.799795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:53.065020 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:53.064903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:53.164477 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:53.164431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:53.938303 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:53.938267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:54.453717 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:54.453688    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:27:54.897589 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:54.897551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:55.473145 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:55.473110    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:55.831850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:55.831757    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:56.464716 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:56.464686    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:27:56.745241 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:56.745114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:57.181245 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:57.181184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:57.574425 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:57.574334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:58.095083 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.095044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:58.410320 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.410279    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"]
Sep 11 18:27:58.413977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.413955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.416063 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.416039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1de19d-serving-cert\""
Sep 11 18:27:58.428222 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.425302    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"]
Sep 11 18:27:58.433537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.433369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzn27\" (UniqueName: \"kubernetes.io/projected/84338234-a45a-4d8e-b181-f4473dadcf99-kube-api-access-jzn27\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.433537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.433426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-maas-api-tls\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.433537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.433470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-metrics-certs\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.433537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.433507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84338234-a45a-4d8e-b181-f4473dadcf99-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.523716 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.523672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:58.534582 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.534538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-metrics-certs\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.534582 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.534585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84338234-a45a-4d8e-b181-f4473dadcf99-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.534843 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.534654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzn27\" (UniqueName: \"kubernetes.io/projected/84338234-a45a-4d8e-b181-f4473dadcf99-kube-api-access-jzn27\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.534843 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.534680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-maas-api-tls\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.534843 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:27:58.534791    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-serving-cert: secret "maas-api-e2e-inf-b-1de19d-serving-cert" not found
Sep 11 18:27:58.534988 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:27:58.534851    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-maas-api-tls podName:84338234-a45a-4d8e-b181-f4473dadcf99 nodeName:}" failed. No retries permitted until 2026-09-11 18:27:59.034835446 +0000 UTC m=+760.131429055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-maas-api-tls") pod "maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp" (UID: "84338234-a45a-4d8e-b181-f4473dadcf99") : secret "maas-api-e2e-inf-b-1de19d-serving-cert" not found
Sep 11 18:27:58.535371 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.535344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84338234-a45a-4d8e-b181-f4473dadcf99-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.537150 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.537126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-metrics-certs\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.545850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.545821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzn27\" (UniqueName: \"kubernetes.io/projected/84338234-a45a-4d8e-b181-f4473dadcf99-kube-api-access-jzn27\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:58.629040 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.628987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:58.725038 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:58.724929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:59.039730 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:59.039623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-maas-api-tls\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:59.042092 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:59.042060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-maas-api-tls\") pod \"maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:59.099676 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:59.099636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:59.324695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:59.324595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:27:59.375221 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:59.374807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:59.476225 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:59.475428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:27:59.631037 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:59.631008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"]
Sep 11 18:27:59.633632 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:27:59.633601    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84338234_a45a_4d8e_b181_f4473dadcf99.slice/crio-62de2b0e703e9b860976c38fe5fffdefb37cf7e2b7e31283632668e64a1c48f6 WatchSource:0}: Error finding container 62de2b0e703e9b860976c38fe5fffdefb37cf7e2b7e31283632668e64a1c48f6: Status 404 returned error can't find the container with id 62de2b0e703e9b860976c38fe5fffdefb37cf7e2b7e31283632668e64a1c48f6
Sep 11 18:27:59.859505 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:27:59.859465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:00.504852 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:00.504809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp" event={"ID":"84338234-a45a-4d8e-b181-f4473dadcf99","Type":"ContainerStarted","Data":"daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921"}
Sep 11 18:28:00.505430 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:00.504867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp" event={"ID":"84338234-a45a-4d8e-b181-f4473dadcf99","Type":"ContainerStarted","Data":"62de2b0e703e9b860976c38fe5fffdefb37cf7e2b7e31283632668e64a1c48f6"}
Sep 11 18:28:00.505430 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:00.504942    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:28:00.521852 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:00.521791    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp" podStartSLOduration=2.121343786 podStartE2EDuration="2.521774185s" totalImagesPullingTime="400.430399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:00.520285913 +0000 UTC m=+761.616879545" watchObservedRunningTime="2026-09-11 18:28:00.521774185 +0000 UTC m=+761.618367815"
Sep 11 18:28:00.787743 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:00.787651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:01.674966 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:01.674910    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:02.925455 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:02.925414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:03.689559 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:03.689505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:03.752868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:03.752819    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:04.854483 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:04.854372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:05.565971 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:05.565918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:06.342752 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:06.342718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:06.515330 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:06.515297    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:28:07.246620 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:07.246582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:07.594544 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:07.594508    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": read tcp 10.133.0.2:45128->10.133.0.27:15021: read: connection reset by peer" start-of-body=
Sep 11 18:28:07.594984 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:07.594951    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": read tcp 10.133.0.2:45128->10.133.0.27:15021: read: connection reset by peer"
Sep 11 18:28:07.979795 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:07.979752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:08.547246 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:08.547187    2580 generic.go:356] "Generic (PLEG): container finished" podID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerID="23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e" exitCode=0
Sep 11 18:28:08.547431 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:08.547257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" event={"ID":"d49cbaa6-1faa-45ed-8577-f474aeb73a9b","Type":"ContainerDied","Data":"23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e"}
Sep 11 18:28:08.547712 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:08.547694    2580 scope.go:117] "RemoveContainer" containerID="23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e"
Sep 11 18:28:08.548857 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:08.548821    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:28:08.548957 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:08.548918    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:28:08.549023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:08.548967    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:28:08.852847 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:08.852746    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:09.554443 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:09.554401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" event={"ID":"d49cbaa6-1faa-45ed-8577-f474aeb73a9b","Type":"ContainerStarted","Data":"791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56"}
Sep 11 18:28:09.635817 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:09.635773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:10.353053 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:10.353016    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:10.542567 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:10.542529    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:28:10.542799 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:10.542780    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:28:10.542846 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:10.542824    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 18:28:11.175180 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:11.175140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:11.542630 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:11.542537    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:28:11.542630 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:11.542606    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 18:28:12.542472 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:12.542426    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:28:12.542660 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:12.542486    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 18:28:13.542715 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:13.542674    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:28:13.543092 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:13.542737    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 18:28:14.341396 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.341355    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"]
Sep 11 18:28:14.341791 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.341756    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv" podUID="e3aa4c5e-2db5-46c5-910a-3175973c0c81" containerName="maas-api" containerID="cri-o://3119bbc00813e43b0240141d94079b80b19e200023da271d46edf8bf51502633" gracePeriod=30
Sep 11 18:28:14.375730 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.375438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:14.478606 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.478403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:14.535040 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.534982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:14.542864 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.542787    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:28:14.543345 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.542858    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 18:28:14.588508 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.588457    2580 generic.go:356] "Generic (PLEG): container finished" podID="e3aa4c5e-2db5-46c5-910a-3175973c0c81" containerID="3119bbc00813e43b0240141d94079b80b19e200023da271d46edf8bf51502633" exitCode=0
Sep 11 18:28:14.588508 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.588491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv" event={"ID":"e3aa4c5e-2db5-46c5-910a-3175973c0c81","Type":"ContainerDied","Data":"3119bbc00813e43b0240141d94079b80b19e200023da271d46edf8bf51502633"}
Sep 11 18:28:14.635210 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.635154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:14.638422 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.638399    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:28:14.691533 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.691500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-metrics-certs\") pod \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") "
Sep 11 18:28:14.691708 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.691553    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8tnb\" (UniqueName: \"kubernetes.io/projected/e3aa4c5e-2db5-46c5-910a-3175973c0c81-kube-api-access-l8tnb\") pod \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") "
Sep 11 18:28:14.691708 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.691665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3aa4c5e-2db5-46c5-910a-3175973c0c81-openshift-service-ca\") pod \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") "
Sep 11 18:28:14.691708 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.691688    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-maas-api-tls\") pod \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\" (UID: \"e3aa4c5e-2db5-46c5-910a-3175973c0c81\") "
Sep 11 18:28:14.692048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.692018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3aa4c5e-2db5-46c5-910a-3175973c0c81-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3aa4c5e-2db5-46c5-910a-3175973c0c81" (UID: "e3aa4c5e-2db5-46c5-910a-3175973c0c81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:14.693957 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.693925    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3aa4c5e-2db5-46c5-910a-3175973c0c81" (UID: "e3aa4c5e-2db5-46c5-910a-3175973c0c81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:28:14.693957 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.693945    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3aa4c5e-2db5-46c5-910a-3175973c0c81" (UID: "e3aa4c5e-2db5-46c5-910a-3175973c0c81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:28:14.694099 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.694083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aa4c5e-2db5-46c5-910a-3175973c0c81-kube-api-access-l8tnb" (OuterVolumeSpecName: "kube-api-access-l8tnb") pod "e3aa4c5e-2db5-46c5-910a-3175973c0c81" (UID: "e3aa4c5e-2db5-46c5-910a-3175973c0c81"). InnerVolumeSpecName "kube-api-access-l8tnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:14.727319 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.727255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:14.793074 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.793035    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3aa4c5e-2db5-46c5-910a-3175973c0c81-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:14.793074 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.793072    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:14.793364 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.793087    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3aa4c5e-2db5-46c5-910a-3175973c0c81-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:14.793364 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.793095    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8tnb\" (UniqueName: \"kubernetes.io/projected/e3aa4c5e-2db5-46c5-910a-3175973c0c81-kube-api-access-l8tnb\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:14.804087 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:14.804039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:15.195338 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.195297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:15.284152 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.284107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:15.429100 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.429064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:15.542601 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.542501    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:28:15.542601 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.542567    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 18:28:15.594021 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.593991    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"
Sep 11 18:28:15.594498 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.594025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv" event={"ID":"e3aa4c5e-2db5-46c5-910a-3175973c0c81","Type":"ContainerDied","Data":"0db74ef380eeb76b33e71c07d931197ba26e9296a279bb17114ba14df941428f"}
Sep 11 18:28:15.594498 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.594082    2580 scope.go:117] "RemoveContainer" containerID="3119bbc00813e43b0240141d94079b80b19e200023da271d46edf8bf51502633"
Sep 11 18:28:15.620456 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.620422    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"]
Sep 11 18:28:15.624633 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.624606    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8370956b-f9dc4cdcb-ksvlv"]
Sep 11 18:28:15.710332 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.710296    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"]
Sep 11 18:28:15.710691 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.710665    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3aa4c5e-2db5-46c5-910a-3175973c0c81" containerName="maas-api"
Sep 11 18:28:15.710691 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.710686    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3aa4c5e-2db5-46c5-910a-3175973c0c81" containerName="maas-api"
Sep 11 18:28:15.710881 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.710788    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3aa4c5e-2db5-46c5-910a-3175973c0c81" containerName="maas-api"
Sep 11 18:28:15.715516 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.715487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.717779 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.717753    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-839ba106-gw-openshift-default-dockercfg-kdlwz\""
Sep 11 18:28:15.727226 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.727155    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"]
Sep 11 18:28:15.742720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.742667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:15.802035 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.801943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-socket\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.802035 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.802005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-token\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.802255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.802048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-certs\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.802255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.802072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqk7\" (UniqueName: \"kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-kube-api-access-bnqk7\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.802255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.802112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-credential-socket\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.802255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.802147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-podinfo\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.802255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.802183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istiod-ca-cert\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.802255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.802233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-data\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.802481 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.802464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-envoy\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.903667 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.903625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-credential-socket\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.903874 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.903672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-podinfo\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.903874 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.903721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istiod-ca-cert\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.903874 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.903758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-data\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.903874 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.903798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-envoy\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.903874 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.903836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-socket\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.904125 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.903878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-token\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.904125 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.904004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-certs\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.904125 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.904057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqk7\" (UniqueName: \"kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-kube-api-access-bnqk7\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.904320 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.904236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-data\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.904496 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.904474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-certs\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.904567 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.904527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-socket\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.904603 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.904564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-credential-socket\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.904638 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.904600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istiod-ca-cert\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.906353 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.906324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-envoy\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.906456 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.906331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-podinfo\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.912680 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.912645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-token\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:15.912803 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:15.912658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqk7\" (UniqueName: \"kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-kube-api-access-bnqk7\") pod \"e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") " pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:16.026081 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.025982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:16.046792 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.046690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:16.311647 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.311618    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"]
Sep 11 18:28:16.313609 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:28:16.313579    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615dcf61_cdd6_4110_a06f_1af5d2fcba97.slice/crio-db9c2a10a3878ef35d55ff89cc23667027377c6406f098d882eb06a698872f91 WatchSource:0}: Error finding container db9c2a10a3878ef35d55ff89cc23667027377c6406f098d882eb06a698872f91: Status 404 returned error can't find the container with id db9c2a10a3878ef35d55ff89cc23667027377c6406f098d882eb06a698872f91
Sep 11 18:28:16.315842 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.315802    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:28:16.315924 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.315874    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:28:16.315924 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.315905    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:28:16.515491 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.515358    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:16.547559 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.547530    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:28:16.548108 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.547927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:28:16.549017 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.548996    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:28:16.593129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.593094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:16.604738 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.604418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" event={"ID":"615dcf61-cdd6-4110-a06f-1af5d2fcba97","Type":"ContainerStarted","Data":"4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e"}
Sep 11 18:28:16.604738 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.604484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" event={"ID":"615dcf61-cdd6-4110-a06f-1af5d2fcba97","Type":"ContainerStarted","Data":"db9c2a10a3878ef35d55ff89cc23667027377c6406f098d882eb06a698872f91"}
Sep 11 18:28:16.628837 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.628781    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" podStartSLOduration=1.6287595609999999 podStartE2EDuration="1.628759561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:16.626453852 +0000 UTC m=+777.723047503" watchObservedRunningTime="2026-09-11 18:28:16.628759561 +0000 UTC m=+777.725353192"
Sep 11 18:28:16.674874 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.674837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:16.801407 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:16.801370    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:17.027148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:17.027107    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:17.032473 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:17.032442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:17.222343 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:17.222282    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:17.395192 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:17.395156    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3aa4c5e-2db5-46c5-910a-3175973c0c81" path="/var/lib/kubelet/pods/e3aa4c5e-2db5-46c5-910a-3175973c0c81/volumes"
Sep 11 18:28:17.611114 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:17.611083    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:17.611990 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:17.611964    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:18.050360 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:18.050323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:18.170966 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:18.170934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:18.276593 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:18.276553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:18.840954 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:18.840916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:19.639985 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:19.639952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:20.369934 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.369894    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"]
Sep 11 18:28:20.374855 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.374827    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.376999 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.376974    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-839ba106-serving-cert\""
Sep 11 18:28:20.387351 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.387320    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"]
Sep 11 18:28:20.431901 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.431870    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:20.446252 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.446211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f93d31c-a94a-4e4e-b975-07f6ca45428a-openshift-service-ca\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.446398 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.446299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-maas-api-tls\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.446398 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.446334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-metrics-certs\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.446398 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.446351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t89n\" (UniqueName: \"kubernetes.io/projected/6f93d31c-a94a-4e4e-b975-07f6ca45428a-kube-api-access-6t89n\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.531407 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.531367    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:20.547644 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.547607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-maas-api-tls\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.547814 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.547657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-metrics-certs\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.547814 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:28:20.547749    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-serving-cert: secret "maas-api-e2e-migrate-839ba106-serving-cert" not found
Sep 11 18:28:20.547814 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.547767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t89n\" (UniqueName: \"kubernetes.io/projected/6f93d31c-a94a-4e4e-b975-07f6ca45428a-kube-api-access-6t89n\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.547814 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:28:20.547811    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-maas-api-tls podName:6f93d31c-a94a-4e4e-b975-07f6ca45428a nodeName:}" failed. No retries permitted until 2026-09-11 18:28:21.047793877 +0000 UTC m=+782.144387486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-maas-api-tls") pod "maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt" (UID: "6f93d31c-a94a-4e4e-b975-07f6ca45428a") : secret "maas-api-e2e-migrate-839ba106-serving-cert" not found
Sep 11 18:28:20.547981 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.547936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f93d31c-a94a-4e4e-b975-07f6ca45428a-openshift-service-ca\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.548591 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.548570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f93d31c-a94a-4e4e-b975-07f6ca45428a-openshift-service-ca\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.550166 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.550144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-metrics-certs\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.558769 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.558742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t89n\" (UniqueName: \"kubernetes.io/projected/6f93d31c-a94a-4e4e-b975-07f6ca45428a-kube-api-access-6t89n\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:20.668694 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:20.668633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:21.021460 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:21.021368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:21.054212 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:21.054130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-maas-api-tls\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:21.056774 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:21.056742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-maas-api-tls\") pod \"maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:21.119559 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:21.119517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:21.286686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:21.286582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:21.562794 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:21.562554    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"]
Sep 11 18:28:21.565402 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:28:21.565366    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f93d31c_a94a_4e4e_b975_07f6ca45428a.slice/crio-e664e298ac0411d7efdaaf4f8138f79daa22d31dc7b1eaa33adad0c7754d3764 WatchSource:0}: Error finding container e664e298ac0411d7efdaaf4f8138f79daa22d31dc7b1eaa33adad0c7754d3764: Status 404 returned error can't find the container with id e664e298ac0411d7efdaaf4f8138f79daa22d31dc7b1eaa33adad0c7754d3764
Sep 11 18:28:21.636772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:21.636739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt" event={"ID":"6f93d31c-a94a-4e4e-b975-07f6ca45428a","Type":"ContainerStarted","Data":"e664e298ac0411d7efdaaf4f8138f79daa22d31dc7b1eaa33adad0c7754d3764"}
Sep 11 18:28:21.640568 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:21.640523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:22.463186 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:22.463140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:22.643529 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:22.643490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt" event={"ID":"6f93d31c-a94a-4e4e-b975-07f6ca45428a","Type":"ContainerStarted","Data":"fef5603a6742bdb45f8f1867c802c2ac79b747716988ee474420551cb81d3858"}
Sep 11 18:28:22.643913 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:22.643619    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:22.661274 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:22.661215    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt" podStartSLOduration=2.309882919 podStartE2EDuration="2.661183975s" totalImagesPullingTime="351.301056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:22.659054228 +0000 UTC m=+783.755647858" watchObservedRunningTime="2026-09-11 18:28:22.661183975 +0000 UTC m=+783.757777607"
Sep 11 18:28:23.084368 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:23.084330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:23.184468 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:23.184421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:24.062513 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:24.062476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:24.267408 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:24.267369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:24.363482 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:24.363396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:24.463261 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:24.463229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:24.604531 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:24.604486    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:24.862947 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:24.862910    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:28.653724 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:28.653686    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:30.455786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:30.455752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:30.556605 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:30.556567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:42.695517 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:42.695471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:42.742561 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:42.742467    2580 generic.go:356] "Generic (PLEG): container finished" podID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerID="4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e" exitCode=0
Sep 11 18:28:42.742561 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:42.742520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" event={"ID":"615dcf61-cdd6-4110-a06f-1af5d2fcba97","Type":"ContainerDied","Data":"4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e"}
Sep 11 18:28:42.742994 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:42.742976    2580 scope.go:117] "RemoveContainer" containerID="4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e"
Sep 11 18:28:42.744348 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:42.744318    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:28:42.744424 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:42.744390    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:28:42.744424 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:42.744421    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:28:43.661010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:43.660962    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:43.751919 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:43.751869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" event={"ID":"615dcf61-cdd6-4110-a06f-1af5d2fcba97","Type":"ContainerStarted","Data":"0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7"}
Sep 11 18:28:44.027076 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:44.026976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:44.028265 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:44.028230    2580 patch_prober.go:28] interesting pod/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 11 18:28:44.028383 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:44.028308    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 11 18:28:44.598015 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:44.597974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:45.027374 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:45.027332    2580 patch_prober.go:28] interesting pod/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 11 18:28:45.027773 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:45.027398    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 11 18:28:45.361019 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:45.360921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:46.026622 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:46.026585    2580 patch_prober.go:28] interesting pod/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 11 18:28:46.026817 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:46.026653    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 11 18:28:46.160114 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:46.160077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:47.027143 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:47.027104    2580 patch_prober.go:28] interesting pod/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 11 18:28:47.027360 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:47.027168    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 11 18:28:47.062592 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:47.062543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:47.611580 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:47.611540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:47.915878 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:47.915840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:48.026994 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:48.026956    2580 patch_prober.go:28] interesting pod/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 11 18:28:48.027219 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:48.027026    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 11 18:28:48.893361 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:48.893327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:49.031175 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.031144    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:49.032304 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.032281    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:49.594000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.593954    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"]
Sep 11 18:28:49.594293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.594242    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt" podUID="6f93d31c-a94a-4e4e-b975-07f6ca45428a" containerName="maas-api" containerID="cri-o://fef5603a6742bdb45f8f1867c802c2ac79b747716988ee474420551cb81d3858" gracePeriod=30
Sep 11 18:28:49.625486 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.625437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:49.716004 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.715964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:49.773282 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.773086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:49.798334 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.798281    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f93d31c-a94a-4e4e-b975-07f6ca45428a" containerID="fef5603a6742bdb45f8f1867c802c2ac79b747716988ee474420551cb81d3858" exitCode=0
Sep 11 18:28:49.798582 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.798346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt" event={"ID":"6f93d31c-a94a-4e4e-b975-07f6ca45428a","Type":"ContainerDied","Data":"fef5603a6742bdb45f8f1867c802c2ac79b747716988ee474420551cb81d3858"}
Sep 11 18:28:49.857121 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.857019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:49.858781 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.858762    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:49.920627 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.920589    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f93d31c-a94a-4e4e-b975-07f6ca45428a-openshift-service-ca\") pod \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") "
Sep 11 18:28:49.921099 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.920675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t89n\" (UniqueName: \"kubernetes.io/projected/6f93d31c-a94a-4e4e-b975-07f6ca45428a-kube-api-access-6t89n\") pod \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") "
Sep 11 18:28:49.921099 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.920704    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-maas-api-tls\") pod \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") "
Sep 11 18:28:49.921099 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.920731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-metrics-certs\") pod \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\" (UID: \"6f93d31c-a94a-4e4e-b975-07f6ca45428a\") "
Sep 11 18:28:49.921099 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.921042    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f93d31c-a94a-4e4e-b975-07f6ca45428a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f93d31c-a94a-4e4e-b975-07f6ca45428a" (UID: "6f93d31c-a94a-4e4e-b975-07f6ca45428a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:49.923000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.922970    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f93d31c-a94a-4e4e-b975-07f6ca45428a" (UID: "6f93d31c-a94a-4e4e-b975-07f6ca45428a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:28:49.923119 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.923003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6f93d31c-a94a-4e4e-b975-07f6ca45428a" (UID: "6f93d31c-a94a-4e4e-b975-07f6ca45428a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:28:49.923119 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.923084    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f93d31c-a94a-4e4e-b975-07f6ca45428a-kube-api-access-6t89n" (OuterVolumeSpecName: "kube-api-access-6t89n") pod "6f93d31c-a94a-4e4e-b975-07f6ca45428a" (UID: "6f93d31c-a94a-4e4e-b975-07f6ca45428a"). InnerVolumeSpecName "kube-api-access-6t89n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:49.929759 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.929713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:49.986652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.986607    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"]
Sep 11 18:28:49.993164 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:49.993111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:50.022001 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.021961    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t89n\" (UniqueName: \"kubernetes.io/projected/6f93d31c-a94a-4e4e-b975-07f6ca45428a-kube-api-access-6t89n\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:50.022001 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.021992    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:50.022001 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.022005    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f93d31c-a94a-4e4e-b975-07f6ca45428a-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:50.022274 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.022019    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f93d31c-a94a-4e4e-b975-07f6ca45428a-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:50.336129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.336087    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:50.437940 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.437898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:50.740510 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.740476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:50.811121 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.811082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt" event={"ID":"6f93d31c-a94a-4e4e-b975-07f6ca45428a","Type":"ContainerDied","Data":"e664e298ac0411d7efdaaf4f8138f79daa22d31dc7b1eaa33adad0c7754d3764"}
Sep 11 18:28:50.811121 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.811116    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"
Sep 11 18:28:50.811391 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.811183    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy" containerID="cri-o://0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7" gracePeriod=30
Sep 11 18:28:50.811391 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.811181    2580 scope.go:117] "RemoveContainer" containerID="fef5603a6742bdb45f8f1867c802c2ac79b747716988ee474420551cb81d3858"
Sep 11 18:28:50.835130 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.835091    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:50.846702 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.846664    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"]
Sep 11 18:28:50.850927 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:50.850902    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-839ba106-5f65766fdc-4drmt"]
Sep 11 18:28:51.205231 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:51.205180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:51.396034 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:51.396002    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f93d31c-a94a-4e4e-b975-07f6ca45428a" path="/var/lib/kubelet/pods/6f93d31c-a94a-4e4e-b975-07f6ca45428a/volumes"
Sep 11 18:28:51.931129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:51.931091    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:52.834774 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:52.834717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:52.975162 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:52.975121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:53.074365 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:53.074324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:53.721069 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:53.721027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:54.241898 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:54.241866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:54.567011 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:54.566924    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:55.515589 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.515554    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"]
Sep 11 18:28:55.516010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.515990    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f93d31c-a94a-4e4e-b975-07f6ca45428a" containerName="maas-api"
Sep 11 18:28:55.516010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.516014    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f93d31c-a94a-4e4e-b975-07f6ca45428a" containerName="maas-api"
Sep 11 18:28:55.516138 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.516123    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f93d31c-a94a-4e4e-b975-07f6ca45428a" containerName="maas-api"
Sep 11 18:28:55.520852 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.520814    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.522872 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.522844    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3565479f-serving-cert\""
Sep 11 18:28:55.532983 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.532949    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"]
Sep 11 18:28:55.551381 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.551347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:55.574385 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.574348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-metrics-certs\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.574591 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.574483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec332bed-73de-431d-a59d-738e57d8e632-openshift-service-ca\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.574591 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.574548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-maas-api-tls\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.574591 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.574569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zjcf\" (UniqueName: \"kubernetes.io/projected/ec332bed-73de-431d-a59d-738e57d8e632-kube-api-access-2zjcf\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.651051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.651012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:55.675345 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.675298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-metrics-certs\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.675467 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.675403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec332bed-73de-431d-a59d-738e57d8e632-openshift-service-ca\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.675467 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.675437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-maas-api-tls\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.675467 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.675456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zjcf\" (UniqueName: \"kubernetes.io/projected/ec332bed-73de-431d-a59d-738e57d8e632-kube-api-access-2zjcf\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.676327 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.676294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec332bed-73de-431d-a59d-738e57d8e632-openshift-service-ca\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.677836 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.677809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-maas-api-tls\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.677941 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.677845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-metrics-certs\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.684647 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.684626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zjcf\" (UniqueName: \"kubernetes.io/projected/ec332bed-73de-431d-a59d-738e57d8e632-kube-api-access-2zjcf\") pod \"maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:55.743852 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.743801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:55.834170 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:55.834142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:56.055094 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.055062    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:56.078280 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-socket\") pod \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") "
Sep 11 18:28:56.078280 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078232    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnqk7\" (UniqueName: \"kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-kube-api-access-bnqk7\") pod \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") "
Sep 11 18:28:56.078511 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078285    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-credential-socket\") pod \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") "
Sep 11 18:28:56.078511 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-envoy\") pod \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") "
Sep 11 18:28:56.078511 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-certs\") pod \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") "
Sep 11 18:28:56.078511 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078341    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istiod-ca-cert\") pod \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") "
Sep 11 18:28:56.078511 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-podinfo\") pod \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") "
Sep 11 18:28:56.078511 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-token\") pod \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") "
Sep 11 18:28:56.078511 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078451    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-data\") pod \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\" (UID: \"615dcf61-cdd6-4110-a06f-1af5d2fcba97\") "
Sep 11 18:28:56.078877 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078642    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "615dcf61-cdd6-4110-a06f-1af5d2fcba97" (UID: "615dcf61-cdd6-4110-a06f-1af5d2fcba97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:28:56.078877 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078657    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "615dcf61-cdd6-4110-a06f-1af5d2fcba97" (UID: "615dcf61-cdd6-4110-a06f-1af5d2fcba97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:28:56.078877 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078845    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-credential-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:56.078877 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078853    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "615dcf61-cdd6-4110-a06f-1af5d2fcba97" (UID: "615dcf61-cdd6-4110-a06f-1af5d2fcba97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:28:56.078877 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078866    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:56.078877 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-data" (OuterVolumeSpecName: "istio-data") pod "615dcf61-cdd6-4110-a06f-1af5d2fcba97" (UID: "615dcf61-cdd6-4110-a06f-1af5d2fcba97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:28:56.079150 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.078972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "615dcf61-cdd6-4110-a06f-1af5d2fcba97" (UID: "615dcf61-cdd6-4110-a06f-1af5d2fcba97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:56.081108 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.081075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "615dcf61-cdd6-4110-a06f-1af5d2fcba97" (UID: "615dcf61-cdd6-4110-a06f-1af5d2fcba97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:28:56.081715 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.081686    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-token" (OuterVolumeSpecName: "istio-token") pod "615dcf61-cdd6-4110-a06f-1af5d2fcba97" (UID: "615dcf61-cdd6-4110-a06f-1af5d2fcba97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:56.081875 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.081790    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "615dcf61-cdd6-4110-a06f-1af5d2fcba97" (UID: "615dcf61-cdd6-4110-a06f-1af5d2fcba97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:28:56.082096 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.082068    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-kube-api-access-bnqk7" (OuterVolumeSpecName: "kube-api-access-bnqk7") pod "615dcf61-cdd6-4110-a06f-1af5d2fcba97" (UID: "615dcf61-cdd6-4110-a06f-1af5d2fcba97"). InnerVolumeSpecName "kube-api-access-bnqk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:56.115703 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.115675    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"]
Sep 11 18:28:56.117759 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:28:56.117730    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec332bed_73de_431d_a59d_738e57d8e632.slice/crio-181d7d11c4034955ebb55776b5d2556edea814f191bfcb758633b68e451d67ee WatchSource:0}: Error finding container 181d7d11c4034955ebb55776b5d2556edea814f191bfcb758633b68e451d67ee: Status 404 returned error can't find the container with id 181d7d11c4034955ebb55776b5d2556edea814f191bfcb758633b68e451d67ee
Sep 11 18:28:56.154819 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.154783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:56.180248 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.180213    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-envoy\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:56.180248 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.180245    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istiod-ca-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:56.180248 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.180255    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-podinfo\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:56.180477 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.180263    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-token\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:56.180477 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.180271    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-istio-data\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:56.180477 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.180278    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615dcf61-cdd6-4110-a06f-1af5d2fcba97-workload-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:56.180477 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.180286    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnqk7\" (UniqueName: \"kubernetes.io/projected/615dcf61-cdd6-4110-a06f-1af5d2fcba97-kube-api-access-bnqk7\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:28:56.254141 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.254095    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:56.573387 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.573352    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:56.841912 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.841867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x" event={"ID":"ec332bed-73de-431d-a59d-738e57d8e632","Type":"ContainerStarted","Data":"53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460"}
Sep 11 18:28:56.842098 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.841941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x" event={"ID":"ec332bed-73de-431d-a59d-738e57d8e632","Type":"ContainerStarted","Data":"181d7d11c4034955ebb55776b5d2556edea814f191bfcb758633b68e451d67ee"}
Sep 11 18:28:56.842098 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.841964    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:28:56.843582 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.843554    2580 generic.go:356] "Generic (PLEG): container finished" podID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerID="0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7" exitCode=0
Sep 11 18:28:56.843697 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.843610    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"
Sep 11 18:28:56.843697 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.843635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" event={"ID":"615dcf61-cdd6-4110-a06f-1af5d2fcba97","Type":"ContainerDied","Data":"0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7"}
Sep 11 18:28:56.843697 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.843679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299" event={"ID":"615dcf61-cdd6-4110-a06f-1af5d2fcba97","Type":"ContainerDied","Data":"db9c2a10a3878ef35d55ff89cc23667027377c6406f098d882eb06a698872f91"}
Sep 11 18:28:56.843814 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.843703    2580 scope.go:117] "RemoveContainer" containerID="0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7"
Sep 11 18:28:56.853319 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.853295    2580 scope.go:117] "RemoveContainer" containerID="4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e"
Sep 11 18:28:56.862020 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.861999    2580 scope.go:117] "RemoveContainer" containerID="0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7"
Sep 11 18:28:56.862382 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:28:56.862352    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7\": container with ID starting with 0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7 not found: ID does not exist" containerID="0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7"
Sep 11 18:28:56.862490 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.862392    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7"} err="failed to get container status \"0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7\": rpc error: code = NotFound desc = could not find container \"0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7\": container with ID starting with 0b9a7ca19031413fb32027a1db0174d10110ca741818c7c985045e8645571ad7 not found: ID does not exist"
Sep 11 18:28:56.862490 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.862413    2580 scope.go:117] "RemoveContainer" containerID="4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e"
Sep 11 18:28:56.862716 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:28:56.862701    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e\": container with ID starting with 4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e not found: ID does not exist" containerID="4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e"
Sep 11 18:28:56.862759 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.862720    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e"} err="failed to get container status \"4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e\": rpc error: code = NotFound desc = could not find container \"4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e\": container with ID starting with 4dbe7f5fdee5911fb6926b743d1da14b1e15a97ba6f8d2139bcdb86b5bf5791e not found: ID does not exist"
Sep 11 18:28:56.864949 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.864904    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x" podStartSLOduration=1.453511917 podStartE2EDuration="1.864889413s" totalImagesPullingTime="411.377496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:56.863731602 +0000 UTC m=+817.960325231" watchObservedRunningTime="2026-09-11 18:28:56.864889413 +0000 UTC m=+817.961483043"
Sep 11 18:28:56.881954 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.881911    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"]
Sep 11 18:28:56.888679 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:56.888644    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-839ba106-gw-openshift-default-79f44b5d5b-g5299"]
Sep 11 18:28:57.347250 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:57.347194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:57.397579 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:57.397531    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" path="/var/lib/kubelet/pods/615dcf61-cdd6-4110-a06f-1af5d2fcba97/volumes"
Sep 11 18:28:58.079131 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:58.079088    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:28:58.894823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:28:58.894777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:02.856053 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:02.856020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:29:17.737641 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:17.737589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:18.154860 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.154819    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"]
Sep 11 18:29:18.155342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.155315    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy"
Sep 11 18:29:18.155342 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.155335    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy"
Sep 11 18:29:18.155581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.155359    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy"
Sep 11 18:29:18.155581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.155365    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy"
Sep 11 18:29:18.155581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.155476    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy"
Sep 11 18:29:18.155581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.155491    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="615dcf61-cdd6-4110-a06f-1af5d2fcba97" containerName="istio-proxy"
Sep 11 18:29:18.158901 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.158871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.161371 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.161342    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"openshift-service-ca.crt\""
Sep 11 18:29:18.162029 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.162009    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"simulator-w4-724327-kserve-self-signed-certs\""
Sep 11 18:29:18.162137 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.162030    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"kube-root-ca.crt\""
Sep 11 18:29:18.162184 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.162169    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"default-dockercfg-wrpkw\""
Sep 11 18:29:18.169612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.169582    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"]
Sep 11 18:29:18.183209 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.183173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:18.280335 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.280282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-model-cache\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.280335 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.280337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-dshm\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.280602 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.280439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-tmp-dir\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.280602 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.280544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-home\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.280602 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.280571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1851c24f-43b2-48be-bcba-8471fa8785af-tls-certs\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.280752 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.280612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dqr\" (UniqueName: \"kubernetes.io/projected/1851c24f-43b2-48be-bcba-8471fa8785af-kube-api-access-24dqr\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.381478 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.381428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-dshm\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.381671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.381492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-tmp-dir\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.381671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.381531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-home\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.381671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.381546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1851c24f-43b2-48be-bcba-8471fa8785af-tls-certs\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.381671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.381577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dqr\" (UniqueName: \"kubernetes.io/projected/1851c24f-43b2-48be-bcba-8471fa8785af-kube-api-access-24dqr\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.381671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.381635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-model-cache\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.382010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.381973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-home\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.382155 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.382042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-tmp-dir\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.382155 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.382066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-model-cache\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.383964 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.383930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-dshm\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.384285 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.384266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1851c24f-43b2-48be-bcba-8471fa8785af-tls-certs\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.390791 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.390760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dqr\" (UniqueName: \"kubernetes.io/projected/1851c24f-43b2-48be-bcba-8471fa8785af-kube-api-access-24dqr\") pod \"simulator-w4-724327-kserve-66b98d66f6-kj2nr\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") " pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.470882 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.470768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:18.752802 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.752711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"]
Sep 11 18:29:18.756408 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:29:18.756370    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1851c24f_43b2_48be_bcba_8471fa8785af.slice/crio-fdeaa2b8ba186592a74e89d7cfb73c6da8473f0da098559e2efbf63e7daf2c3e WatchSource:0}: Error finding container fdeaa2b8ba186592a74e89d7cfb73c6da8473f0da098559e2efbf63e7daf2c3e: Status 404 returned error can't find the container with id fdeaa2b8ba186592a74e89d7cfb73c6da8473f0da098559e2efbf63e7daf2c3e
Sep 11 18:29:18.960597 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.960562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr" event={"ID":"1851c24f-43b2-48be-bcba-8471fa8785af","Type":"ContainerStarted","Data":"1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b"}
Sep 11 18:29:18.960775 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.960605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr" event={"ID":"1851c24f-43b2-48be-bcba-8471fa8785af","Type":"ContainerStarted","Data":"fdeaa2b8ba186592a74e89d7cfb73c6da8473f0da098559e2efbf63e7daf2c3e"}
Sep 11 18:29:18.977693 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:18.977616    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr" podStartSLOduration=0.977592659 podStartE2EDuration="977.592659ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:18.975839422 +0000 UTC m=+840.072433054" watchObservedRunningTime="2026-09-11 18:29:18.977592659 +0000 UTC m=+840.074186351"
Sep 11 18:29:19.019820 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:19.019715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:19.942661 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:19.942617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:20.758833 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:20.758761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:21.647519 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:21.647480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:22.361844 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:22.361805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:23.066190 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:23.066144    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:23.953643 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:23.953605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:24.657424 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:24.657380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:25.243299 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.243259    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"]
Sep 11 18:29:25.243801 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.243762    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x" podUID="ec332bed-73de-431d-a59d-738e57d8e632" containerName="maas-api" containerID="cri-o://53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460" gracePeriod=30
Sep 11 18:29:25.276287 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.276252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:25.341999 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.341964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:25.402237 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.402174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:25.492647 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.492595    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:25.507137 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.507057    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:29:25.583997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.583960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:25.653147 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.653113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-maas-api-tls\") pod \"ec332bed-73de-431d-a59d-738e57d8e632\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") "
Sep 11 18:29:25.653376 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.653272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec332bed-73de-431d-a59d-738e57d8e632-openshift-service-ca\") pod \"ec332bed-73de-431d-a59d-738e57d8e632\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") "
Sep 11 18:29:25.653376 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.653301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-metrics-certs\") pod \"ec332bed-73de-431d-a59d-738e57d8e632\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") "
Sep 11 18:29:25.653376 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.653320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zjcf\" (UniqueName: \"kubernetes.io/projected/ec332bed-73de-431d-a59d-738e57d8e632-kube-api-access-2zjcf\") pod \"ec332bed-73de-431d-a59d-738e57d8e632\" (UID: \"ec332bed-73de-431d-a59d-738e57d8e632\") "
Sep 11 18:29:25.653610 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.653581    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec332bed-73de-431d-a59d-738e57d8e632-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec332bed-73de-431d-a59d-738e57d8e632" (UID: "ec332bed-73de-431d-a59d-738e57d8e632"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:25.655488 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.655451    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec332bed-73de-431d-a59d-738e57d8e632" (UID: "ec332bed-73de-431d-a59d-738e57d8e632"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:29:25.655608 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.655494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec332bed-73de-431d-a59d-738e57d8e632" (UID: "ec332bed-73de-431d-a59d-738e57d8e632"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:29:25.655608 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.655518    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec332bed-73de-431d-a59d-738e57d8e632-kube-api-access-2zjcf" (OuterVolumeSpecName: "kube-api-access-2zjcf") pod "ec332bed-73de-431d-a59d-738e57d8e632" (UID: "ec332bed-73de-431d-a59d-738e57d8e632"). InnerVolumeSpecName "kube-api-access-2zjcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:25.754256 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.754176    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec332bed-73de-431d-a59d-738e57d8e632-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:29:25.754256 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.754256    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:29:25.754711 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.754272    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zjcf\" (UniqueName: \"kubernetes.io/projected/ec332bed-73de-431d-a59d-738e57d8e632-kube-api-access-2zjcf\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:29:25.754711 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.754288    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec332bed-73de-431d-a59d-738e57d8e632-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:29:25.915300 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.915252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:25.995309 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.995266    2580 generic.go:356] "Generic (PLEG): container finished" podID="ec332bed-73de-431d-a59d-738e57d8e632" containerID="53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460" exitCode=0
Sep 11 18:29:25.995309 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.995317    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"
Sep 11 18:29:25.995585 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.995347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x" event={"ID":"ec332bed-73de-431d-a59d-738e57d8e632","Type":"ContainerDied","Data":"53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460"}
Sep 11 18:29:25.995585 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.995409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x" event={"ID":"ec332bed-73de-431d-a59d-738e57d8e632","Type":"ContainerDied","Data":"181d7d11c4034955ebb55776b5d2556edea814f191bfcb758633b68e451d67ee"}
Sep 11 18:29:25.995585 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:25.995435    2580 scope.go:117] "RemoveContainer" containerID="53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460"
Sep 11 18:29:26.007495 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:26.007465    2580 scope.go:117] "RemoveContainer" containerID="53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460"
Sep 11 18:29:26.007991 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:29:26.007963    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460\": container with ID starting with 53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460 not found: ID does not exist" containerID="53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460"
Sep 11 18:29:26.008094 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:26.008006    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460"} err="failed to get container status \"53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460\": rpc error: code = NotFound desc = could not find container \"53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460\": container with ID starting with 53a75497aff5a33103b9854cdfad4935e761a4dbfd21ca05f010466176275460 not found: ID does not exist"
Sep 11 18:29:26.021541 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:26.021502    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"]
Sep 11 18:29:26.023395 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:26.023367    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3565479f-65ffb5d88f-lwm8x"]
Sep 11 18:29:26.633560 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:26.633525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:26.801632 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:26.801592    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:27.077591 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:27.077498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:27.385689 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:27.385647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:27.396025 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:27.395994    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec332bed-73de-431d-a59d-738e57d8e632" path="/var/lib/kubelet/pods/ec332bed-73de-431d-a59d-738e57d8e632/volumes"
Sep 11 18:29:27.613302 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:27.613259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:27.714470 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:27.714371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:28.471282 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:28.471235    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:28.471699 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:28.471321    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:28.484592 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:28.484554    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:28.640887 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:28.640813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:29.025097 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:29.025053    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:29:29.639886 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:29.639848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:30.249289 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:30.249253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:30.357129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:30.357089    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:30.679771 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:30.679730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:31.413540 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.413504    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"]
Sep 11 18:29:31.413904 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.413892    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec332bed-73de-431d-a59d-738e57d8e632" containerName="maas-api"
Sep 11 18:29:31.413949 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.413905    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec332bed-73de-431d-a59d-738e57d8e632" containerName="maas-api"
Sep 11 18:29:31.413981 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.413974    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec332bed-73de-431d-a59d-738e57d8e632" containerName="maas-api"
Sep 11 18:29:31.417662 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.417635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.419893 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.419869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-68ac8212-serving-cert\""
Sep 11 18:29:31.429236 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.429189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"]
Sep 11 18:29:31.447576 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.447545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:31.510740 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.510683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-metrics-certs\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.510934 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.510772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tlr\" (UniqueName: \"kubernetes.io/projected/9f167549-326c-4ec6-904d-4b1c2a9376b9-kube-api-access-r5tlr\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.511156 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.511131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f167549-326c-4ec6-904d-4b1c2a9376b9-openshift-service-ca\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.511297 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.511277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-maas-api-tls\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.548052 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.548006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:31.612305 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.612264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f167549-326c-4ec6-904d-4b1c2a9376b9-openshift-service-ca\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.612484 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.612320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-maas-api-tls\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.612484 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.612345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-metrics-certs\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.612484 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.612366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tlr\" (UniqueName: \"kubernetes.io/projected/9f167549-326c-4ec6-904d-4b1c2a9376b9-kube-api-access-r5tlr\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.612676 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:29:31.612500    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-serving-cert: secret "maas-api-e2e-derive-68ac8212-serving-cert" not found
Sep 11 18:29:31.612676 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:29:31.612584    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-maas-api-tls podName:9f167549-326c-4ec6-904d-4b1c2a9376b9 nodeName:}" failed. No retries permitted until 2026-09-11 18:29:32.112558798 +0000 UTC m=+853.209152421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-maas-api-tls") pod "maas-api-e2e-derive-68ac8212-766dfff854-zjgwc" (UID: "9f167549-326c-4ec6-904d-4b1c2a9376b9") : secret "maas-api-e2e-derive-68ac8212-serving-cert" not found
Sep 11 18:29:31.612958 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.612936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f167549-326c-4ec6-904d-4b1c2a9376b9-openshift-service-ca\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.614790 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.614770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-metrics-certs\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.623302 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.623268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tlr\" (UniqueName: \"kubernetes.io/projected/9f167549-326c-4ec6-904d-4b1c2a9376b9-kube-api-access-r5tlr\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:31.689912 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.689774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:31.790979 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:31.790926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:32.049170 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:32.049064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:32.117266 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:32.117221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-maas-api-tls\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:32.119806 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:32.119778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-maas-api-tls\") pod \"maas-api-e2e-derive-68ac8212-766dfff854-zjgwc\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:32.149373 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:32.149338    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:32.329664 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:32.329577    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:32.617922 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:32.617890    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"]
Sep 11 18:29:32.620183 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:29:32.620149    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f167549_326c_4ec6_904d_4b1c2a9376b9.slice/crio-e84c2aa92a7ac4c9f23c22aadb9ab3a85827550ce36af7d6b505f9b3e431aa44 WatchSource:0}: Error finding container e84c2aa92a7ac4c9f23c22aadb9ab3a85827550ce36af7d6b505f9b3e431aa44: Status 404 returned error can't find the container with id e84c2aa92a7ac4c9f23c22aadb9ab3a85827550ce36af7d6b505f9b3e431aa44
Sep 11 18:29:32.644476 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:32.644440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:33.032684 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:33.032639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc" event={"ID":"9f167549-326c-4ec6-904d-4b1c2a9376b9","Type":"ContainerStarted","Data":"6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7"}
Sep 11 18:29:33.033160 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:33.032715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc" event={"ID":"9f167549-326c-4ec6-904d-4b1c2a9376b9","Type":"ContainerStarted","Data":"e84c2aa92a7ac4c9f23c22aadb9ab3a85827550ce36af7d6b505f9b3e431aa44"}
Sep 11 18:29:33.033160 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:33.032745    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:33.050327 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:33.050274    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc" podStartSLOduration=1.718907263 podStartE2EDuration="2.050256101s" totalImagesPullingTime="331.348838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:33.048379093 +0000 UTC m=+854.144972724" watchObservedRunningTime="2026-09-11 18:29:33.050256101 +0000 UTC m=+854.146849732"
Sep 11 18:29:33.460604 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:33.460540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:34.326506 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:34.326470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:35.043542 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:35.043503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:35.780579 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:35.780541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:36.840721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:36.840691    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:39.044480 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:39.044447    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:29:53.835506 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:53.835462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:54.649809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:54.649768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:55.535590 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:55.535552    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:56.253859 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:56.253820    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:56.972744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:56.972707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:57.933696 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:57.933662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:58.645411 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:58.645375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:29:59.430993 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:29:59.430954    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:00.309693 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:00.309630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:01.670139 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:01.670103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:02.255835 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.255798    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"]
Sep 11 18:30:02.256283 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.256042    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc" podUID="9f167549-326c-4ec6-904d-4b1c2a9376b9" containerName="maas-api" containerID="cri-o://6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7" gracePeriod=30
Sep 11 18:30:02.282321 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.282274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:02.436712 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.436666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:02.512908 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.512835    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:30:02.529187 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.529152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:02.597953 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.597908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5tlr\" (UniqueName: \"kubernetes.io/projected/9f167549-326c-4ec6-904d-4b1c2a9376b9-kube-api-access-r5tlr\") pod \"9f167549-326c-4ec6-904d-4b1c2a9376b9\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") "
Sep 11 18:30:02.598160 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.597990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f167549-326c-4ec6-904d-4b1c2a9376b9-openshift-service-ca\") pod \"9f167549-326c-4ec6-904d-4b1c2a9376b9\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") "
Sep 11 18:30:02.598160 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.598040    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-metrics-certs\") pod \"9f167549-326c-4ec6-904d-4b1c2a9376b9\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") "
Sep 11 18:30:02.598160 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.598068    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-maas-api-tls\") pod \"9f167549-326c-4ec6-904d-4b1c2a9376b9\" (UID: \"9f167549-326c-4ec6-904d-4b1c2a9376b9\") "
Sep 11 18:30:02.598466 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.598441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f167549-326c-4ec6-904d-4b1c2a9376b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f167549-326c-4ec6-904d-4b1c2a9376b9" (UID: "9f167549-326c-4ec6-904d-4b1c2a9376b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:02.600325 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.600284    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f167549-326c-4ec6-904d-4b1c2a9376b9" (UID: "9f167549-326c-4ec6-904d-4b1c2a9376b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:30:02.600325 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.600299    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9f167549-326c-4ec6-904d-4b1c2a9376b9" (UID: "9f167549-326c-4ec6-904d-4b1c2a9376b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:30:02.600497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.600414    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f167549-326c-4ec6-904d-4b1c2a9376b9-kube-api-access-r5tlr" (OuterVolumeSpecName: "kube-api-access-r5tlr") pod "9f167549-326c-4ec6-904d-4b1c2a9376b9" (UID: "9f167549-326c-4ec6-904d-4b1c2a9376b9"). InnerVolumeSpecName "kube-api-access-r5tlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:02.627411 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.627366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:02.692395 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.692356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:02.699513 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.699483    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:30:02.699513 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.699516    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f167549-326c-4ec6-904d-4b1c2a9376b9-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:30:02.699660 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.699526    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tlr\" (UniqueName: \"kubernetes.io/projected/9f167549-326c-4ec6-904d-4b1c2a9376b9-kube-api-access-r5tlr\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:30:02.699660 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.699535    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f167549-326c-4ec6-904d-4b1c2a9376b9-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:30:02.931023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:02.930978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:03.124486 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.124449    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"]
Sep 11 18:30:03.124872 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.124845    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f167549-326c-4ec6-904d-4b1c2a9376b9" containerName="maas-api"
Sep 11 18:30:03.124872 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.124866    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f167549-326c-4ec6-904d-4b1c2a9376b9" containerName="maas-api"
Sep 11 18:30:03.125050 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.124954    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f167549-326c-4ec6-904d-4b1c2a9376b9" containerName="maas-api"
Sep 11 18:30:03.127776 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.127757    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.129962 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.129940    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7d7509\"/\"kube-root-ca.crt\""
Sep 11 18:30:03.130067 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.130002    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7d7509\"/\"openshift-service-ca.crt\""
Sep 11 18:30:03.130647 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.130628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7d7509\"/\"test-model-7d7509-kserve-self-signed-certs\""
Sep 11 18:30:03.130740 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.130632    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7d7509\"/\"default-dockercfg-swzrq\""
Sep 11 18:30:03.137699 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.137677    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"]
Sep 11 18:30:03.175819 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.175785    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f167549-326c-4ec6-904d-4b1c2a9376b9" containerID="6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7" exitCode=0
Sep 11 18:30:03.176010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.175833    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"
Sep 11 18:30:03.176010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.175878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc" event={"ID":"9f167549-326c-4ec6-904d-4b1c2a9376b9","Type":"ContainerDied","Data":"6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7"}
Sep 11 18:30:03.176010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.175939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc" event={"ID":"9f167549-326c-4ec6-904d-4b1c2a9376b9","Type":"ContainerDied","Data":"e84c2aa92a7ac4c9f23c22aadb9ab3a85827550ce36af7d6b505f9b3e431aa44"}
Sep 11 18:30:03.176010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.175963    2580 scope.go:117] "RemoveContainer" containerID="6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7"
Sep 11 18:30:03.185022 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.185002    2580 scope.go:117] "RemoveContainer" containerID="6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7"
Sep 11 18:30:03.185324 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:30:03.185300    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7\": container with ID starting with 6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7 not found: ID does not exist" containerID="6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7"
Sep 11 18:30:03.185408 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.185334    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7"} err="failed to get container status \"6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7\": rpc error: code = NotFound desc = could not find container \"6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7\": container with ID starting with 6075f7317fb830ef391e1d9e093d55e56158c92295f183f6fb8a49e2a93bcdf7 not found: ID does not exist"
Sep 11 18:30:03.201419 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.201384    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"]
Sep 11 18:30:03.204085 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.204042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-dshm\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.204264 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.204147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tls-certs\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.204344 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.204266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bwc\" (UniqueName: \"kubernetes.io/projected/dbf989d4-0e9e-45e0-81b5-43e003cf314e-kube-api-access-n4bwc\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.204405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.204325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-home\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.204405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.204369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tmp-dir\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.204511 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.204405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-model-cache\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.205703 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.205682    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68ac8212-766dfff854-zjgwc"]
Sep 11 18:30:03.305617 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.305578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-dshm\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.305617 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.305620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tls-certs\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.305868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.305645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bwc\" (UniqueName: \"kubernetes.io/projected/dbf989d4-0e9e-45e0-81b5-43e003cf314e-kube-api-access-n4bwc\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.305868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.305681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-home\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.305868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.305717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tmp-dir\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.305868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.305746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-model-cache\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.306111 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.306078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-home\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.306172 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.306140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tmp-dir\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.306233 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.306166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-model-cache\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.307883 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.307862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-dshm\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.308147 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.308121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tls-certs\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.319991 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.319960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bwc\" (UniqueName: \"kubernetes.io/projected/dbf989d4-0e9e-45e0-81b5-43e003cf314e-kube-api-access-n4bwc\") pod \"test-model-7d7509-kserve-8567659f6f-k495v\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") " pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.322753 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.322720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:03.395590 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.395552    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f167549-326c-4ec6-904d-4b1c2a9376b9" path="/var/lib/kubelet/pods/9f167549-326c-4ec6-904d-4b1c2a9376b9/volumes"
Sep 11 18:30:03.437647 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.437560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:03.486013 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.485967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:03.586584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.586536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:03.712746 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.712722    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"]
Sep 11 18:30:03.715779 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:30:03.715749    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbf989d4_0e9e_45e0_81b5_43e003cf314e.slice/crio-539f9d1967759c951a9b9f44d70f7f6f7b1471cff341c597ee5caf9520279136 WatchSource:0}: Error finding container 539f9d1967759c951a9b9f44d70f7f6f7b1471cff341c597ee5caf9520279136: Status 404 returned error can't find the container with id 539f9d1967759c951a9b9f44d70f7f6f7b1471cff341c597ee5caf9520279136
Sep 11 18:30:03.850923 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:03.850882    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:04.184272 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:04.184230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v" event={"ID":"dbf989d4-0e9e-45e0-81b5-43e003cf314e","Type":"ContainerStarted","Data":"fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2"}
Sep 11 18:30:04.184272 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:04.184277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v" event={"ID":"dbf989d4-0e9e-45e0-81b5-43e003cf314e","Type":"ContainerStarted","Data":"539f9d1967759c951a9b9f44d70f7f6f7b1471cff341c597ee5caf9520279136"}
Sep 11 18:30:04.202932 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:04.202874    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v" podStartSLOduration=1.202857191 podStartE2EDuration="1.202857191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:04.201250093 +0000 UTC m=+885.297843726" watchObservedRunningTime="2026-09-11 18:30:04.202857191 +0000 UTC m=+885.299450881"
Sep 11 18:30:04.714450 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:04.714406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:04.886630 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:04.886590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:05.455110 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:05.455059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:06.176746 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:06.176715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:06.963722 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:06.963682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:11.091161 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:11.091119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:11.199397 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:11.199356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:11.293031 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:11.292954    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:11.389936 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:11.389884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:11.748470 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:11.748427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:12.797709 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:12.797665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:13.437950 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:13.437908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:13.438161 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:13.437976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:13.450733 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:13.450690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:13.742757 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:13.742638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:14.254240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:14.254186    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:30:14.571713 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:14.571618    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:15.335323 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:15.335284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:16.131550 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:16.131516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:16.953873 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:16.953828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:17.745540 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:17.745490    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:22.996211 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:22.996166    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"]
Sep 11 18:30:22.996627 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:22.996398    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r" podUID="bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa" containerName="maas-api" containerID="cri-o://fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062" gracePeriod=30
Sep 11 18:30:23.097711 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.097681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:23.177496 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.177462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:23.253951 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.253861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:23.258608 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.258579    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:30:23.288650 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.288599    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcd2n\" (UniqueName: \"kubernetes.io/projected/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-kube-api-access-zcd2n\") pod \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") "
Sep 11 18:30:23.288864 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.288687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-openshift-service-ca\") pod \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") "
Sep 11 18:30:23.288864 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.288753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-metrics-certs\") pod \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") "
Sep 11 18:30:23.289455 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.289394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa" (UID: "bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:23.290428 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.290398    2580 generic.go:356] "Generic (PLEG): container finished" podID="bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa" containerID="fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062" exitCode=0
Sep 11 18:30:23.290550 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.290460    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"
Sep 11 18:30:23.290550 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.290486    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r" event={"ID":"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa","Type":"ContainerDied","Data":"fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062"}
Sep 11 18:30:23.290550 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.290548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r" event={"ID":"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa","Type":"ContainerDied","Data":"20ce0ea98ea29f66c223f02bdc77b23ac2f5ac49b947e7713cbbb675169b5768"}
Sep 11 18:30:23.291137 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.290579    2580 scope.go:117] "RemoveContainer" containerID="fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062"
Sep 11 18:30:23.291824 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.291781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-kube-api-access-zcd2n" (OuterVolumeSpecName: "kube-api-access-zcd2n") pod "bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa" (UID: "bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa"). InnerVolumeSpecName "kube-api-access-zcd2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:23.292360 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.292336    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa" (UID: "bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:30:23.312225 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.312181    2580 scope.go:117] "RemoveContainer" containerID="fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062"
Sep 11 18:30:23.312621 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:30:23.312602    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062\": container with ID starting with fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062 not found: ID does not exist" containerID="fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062"
Sep 11 18:30:23.312688 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.312629    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062"} err="failed to get container status \"fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062\": rpc error: code = NotFound desc = could not find container \"fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062\": container with ID starting with fa934831fe24e98d105d24bcc84d44915e6b21dfaccdb3bbee7d4b6955bb2062 not found: ID does not exist"
Sep 11 18:30:23.352112 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.352068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:23.389369 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.389325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-maas-api-tls\") pod \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\" (UID: \"bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa\") "
Sep 11 18:30:23.389648 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.389629    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:30:23.389733 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.389652    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:30:23.389733 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.389667    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcd2n\" (UniqueName: \"kubernetes.io/projected/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-kube-api-access-zcd2n\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:30:23.391582 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.391552    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa" (UID: "bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:30:23.408131 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.408090    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:23.490479 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.490437    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:30:23.606701 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.606604    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"]
Sep 11 18:30:23.613040 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.613002    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7e8ab9-75789cb5f-dbv8r"]
Sep 11 18:30:23.710219 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.710161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:23.814133 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:23.814077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:24.528318 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:24.528285    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:25.308346 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:25.308300    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:25.395375 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:25.395329    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa" path="/var/lib/kubelet/pods/bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa/volumes"
Sep 11 18:30:26.142170 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:26.142136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:26.861522 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:26.861477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:28.788674 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:28.788634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:33.646811 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:33.646774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:37.667593 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:37.667552    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:38.039605 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:38.039512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:43.525650 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:43.525596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:43.626119 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:43.626054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:43.728316 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:43.728278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:30:59.538798 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:30:59.538747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:06.029415 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:06.029375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:07.197016 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:07.196975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:07.298312 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:07.298276    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:07.593280 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:07.593141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:07.679782 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:07.679747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:09.549728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:09.549683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:09.970109 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:09.970074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:10.613249 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:10.613179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:10.713183 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:10.713139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:10.951053 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:10.951015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:11.801565 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:11.801531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:12.577716 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:12.577670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:13.559486 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:13.559438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:14.404984 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.404949    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"]
Sep 11 18:31:14.405404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.405381    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa" containerName="maas-api"
Sep 11 18:31:14.405404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.405405    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa" containerName="maas-api"
Sep 11 18:31:14.405660 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.405513    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3c08e5-b3ce-44ba-be9c-594f7ce9c0aa" containerName="maas-api"
Sep 11 18:31:14.408807 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.408779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.410914 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.410886    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c7c83f-serving-cert\""
Sep 11 18:31:14.422808 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.422769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"]
Sep 11 18:31:14.446708 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.446672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:14.455875 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.455834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-maas-api-tls\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.456098 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.455963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-metrics-certs\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.456249 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.456225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03105077-3a3f-4b8f-b6d3-7660c7d75f44-openshift-service-ca\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.456359 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.456270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txkqv\" (UniqueName: \"kubernetes.io/projected/03105077-3a3f-4b8f-b6d3-7660c7d75f44-kube-api-access-txkqv\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.509519 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.509482    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:14.557587 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.557543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-maas-api-tls\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.557786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.557627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-metrics-certs\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.557786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.557710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03105077-3a3f-4b8f-b6d3-7660c7d75f44-openshift-service-ca\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.557786 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:31:14.557729    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-serving-cert: secret "maas-api-e2e-explicit-c7c83f-serving-cert" not found
Sep 11 18:31:14.557786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.557754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txkqv\" (UniqueName: \"kubernetes.io/projected/03105077-3a3f-4b8f-b6d3-7660c7d75f44-kube-api-access-txkqv\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.557982 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:31:14.557794    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-maas-api-tls podName:03105077-3a3f-4b8f-b6d3-7660c7d75f44 nodeName:}" failed. No retries permitted until 2026-09-11 18:31:15.057773756 +0000 UTC m=+956.154367388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-maas-api-tls") pod "maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp" (UID: "03105077-3a3f-4b8f-b6d3-7660c7d75f44") : secret "maas-api-e2e-explicit-c7c83f-serving-cert" not found
Sep 11 18:31:14.558536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.558498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03105077-3a3f-4b8f-b6d3-7660c7d75f44-openshift-service-ca\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.560163 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.560142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-metrics-certs\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.573474 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.573443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txkqv\" (UniqueName: \"kubernetes.io/projected/03105077-3a3f-4b8f-b6d3-7660c7d75f44-kube-api-access-txkqv\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:14.612354 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.612318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:14.999655 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:14.999620    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:15.061431 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:15.061388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-maas-api-tls\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:15.064116 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:15.064074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-maas-api-tls\") pod \"maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:15.101326 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:15.101285    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:15.321864 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:15.321762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:15.601921 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:15.601878    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"]
Sep 11 18:31:15.604722 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:31:15.604694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03105077_3a3f_4b8f_b6d3_7660c7d75f44.slice/crio-f3a1798c01f35624f7949c34da4d5860e1cb9b5c19b785e4dc94f1b04c1658c3 WatchSource:0}: Error finding container f3a1798c01f35624f7949c34da4d5860e1cb9b5c19b785e4dc94f1b04c1658c3: Status 404 returned error can't find the container with id f3a1798c01f35624f7949c34da4d5860e1cb9b5c19b785e4dc94f1b04c1658c3
Sep 11 18:31:15.606587 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:15.606565    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:31:16.549753 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:16.549720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp" event={"ID":"03105077-3a3f-4b8f-b6d3-7660c7d75f44","Type":"ContainerStarted","Data":"4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e"}
Sep 11 18:31:16.549954 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:16.549780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp" event={"ID":"03105077-3a3f-4b8f-b6d3-7660c7d75f44","Type":"ContainerStarted","Data":"f3a1798c01f35624f7949c34da4d5860e1cb9b5c19b785e4dc94f1b04c1658c3"}
Sep 11 18:31:16.549954 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:16.549921    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:16.568691 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:16.568628    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp" podStartSLOduration=2.072290887 podStartE2EDuration="2.568608026s" totalImagesPullingTime="496.317139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:16.566684138 +0000 UTC m=+957.663277772" watchObservedRunningTime="2026-09-11 18:31:16.568608026 +0000 UTC m=+957.665201656"
Sep 11 18:31:16.752482 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:16.752445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:17.521766 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:17.521725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:22.561293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:22.561260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:31:23.570426 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:23.570387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:23.633738 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:23.633704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:23.725868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:23.725828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:23.826087 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:23.825995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:23.926645 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:23.926603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:24.480522 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:24.480477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:24.571529 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:24.571493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:24.674297 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:24.674252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:26.135488 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:26.135445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:32.077458 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:32.077422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:32.138927 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:32.138889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:32.731901 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:32.731843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:36.631008 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:36.630974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:37.435472 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:37.435425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:38.349705 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:38.349657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:38.441022 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:38.440969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:39.111214 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:39.111161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:39.352053 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:39.352017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:39.452502 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:39.452463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:39.609633 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:39.609594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:39.915002 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:39.914952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:40.140853 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:40.140815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:40.373446 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:40.373407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:40.630926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:40.630811    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:41.151081 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:41.151043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:45.275095 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:45.275046    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:45.506161 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:45.506116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:45.600738 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:45.600646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:46.166516 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:46.166474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:46.968326 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:46.968285    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:47.825404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:47.825371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:48.550080 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:48.550039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:49.316602 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.316562    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"]
Sep 11 18:31:49.320256 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.320229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.322942 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.322912    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3ad8c202-serving-cert\""
Sep 11 18:31:49.333653 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.333617    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"]
Sep 11 18:31:49.350847 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.350814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:49.409962 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.409908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:49.481880 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.481844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-metrics-certs\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.482055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.481885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pqb7\" (UniqueName: \"kubernetes.io/projected/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-kube-api-access-4pqb7\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.482055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.481979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-maas-api-tls\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.482055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.482026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-openshift-service-ca\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.558692 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.558659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:49.582850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.582757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-maas-api-tls\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.582850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.582820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-openshift-service-ca\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.583051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.582897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-metrics-certs\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.583051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.582923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pqb7\" (UniqueName: \"kubernetes.io/projected/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-kube-api-access-4pqb7\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.583637 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.583616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-openshift-service-ca\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.585412 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.585389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-maas-api-tls\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.585530 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.585431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-metrics-certs\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.593509 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.593480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pqb7\" (UniqueName: \"kubernetes.io/projected/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-kube-api-access-4pqb7\") pod \"maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.632630 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.632590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:49.860154 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.860035    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:49.906439 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.906409    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"]
Sep 11 18:31:49.908540 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:31:49.908512    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760e5a91_fb45_49d8_a0a6_9b0b0373a2e4.slice/crio-429af9e9cd75d585d5d4c2305a5b329aa92456356d6350b1e32fbec8be7eae66 WatchSource:0}: Error finding container 429af9e9cd75d585d5d4c2305a5b329aa92456356d6350b1e32fbec8be7eae66: Status 404 returned error can't find the container with id 429af9e9cd75d585d5d4c2305a5b329aa92456356d6350b1e32fbec8be7eae66
Sep 11 18:31:49.957300 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:49.957258    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:50.716031 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:50.715991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2" event={"ID":"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4","Type":"ContainerStarted","Data":"aa034a8db9fed4e52841d78e17c39bfc29f8acf09b75a61b909d3d0092cb2c6c"}
Sep 11 18:31:50.716587 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:50.716050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2" event={"ID":"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4","Type":"ContainerStarted","Data":"429af9e9cd75d585d5d4c2305a5b329aa92456356d6350b1e32fbec8be7eae66"}
Sep 11 18:31:50.716587 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:50.716129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:31:50.733599 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:50.733538    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2" podStartSLOduration=1.3917662800000001 podStartE2EDuration="1.733518589s" totalImagesPullingTime="341.752309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:50.731983999 +0000 UTC m=+991.828577643" watchObservedRunningTime="2026-09-11 18:31:50.733518589 +0000 UTC m=+991.830112220"
Sep 11 18:31:52.633576 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:52.633541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:31:56.729383 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:31:56.729347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:32:11.618290 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:11.618245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:12.541705 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:12.541664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:13.331118 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:13.331081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:14.157827 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:14.157783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:14.926729 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:14.926687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:15.591626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:15.591565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:15.765830 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:15.765794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:16.252244 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:16.252187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:16.343041 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:16.342999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:16.654348 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:16.654302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:16.754886 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:16.754838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:22.865090 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:22.865047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:22.967380 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:22.967321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:23.979566 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:23.979530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:24.848721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:24.848685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:25.568997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:25.568947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:26.380637 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:26.380594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:27.361493 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:27.361451    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:28.137876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:28.137830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:28.909512 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:28.909471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:33.715178 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:33.715145    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"]
Sep 11 18:32:33.715615 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:33.715418    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2" podUID="760e5a91-fb45-49d8-a0a6-9b0b0373a2e4" containerName="maas-api" containerID="cri-o://aa034a8db9fed4e52841d78e17c39bfc29f8acf09b75a61b909d3d0092cb2c6c" gracePeriod=30
Sep 11 18:32:33.738535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:33.738407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:33.811550 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:33.811512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:33.872845 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:33.872803    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:33.928149 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:33.928105    2580 generic.go:356] "Generic (PLEG): container finished" podID="760e5a91-fb45-49d8-a0a6-9b0b0373a2e4" containerID="aa034a8db9fed4e52841d78e17c39bfc29f8acf09b75a61b909d3d0092cb2c6c" exitCode=0
Sep 11 18:32:33.928149 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:33.928152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2" event={"ID":"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4","Type":"ContainerDied","Data":"aa034a8db9fed4e52841d78e17c39bfc29f8acf09b75a61b909d3d0092cb2c6c"}
Sep 11 18:32:33.973280 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:33.973170    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:33.975695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:33.975667    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:32:34.035250 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.035186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:34.088895 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.088851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-openshift-service-ca\") pod \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") "
Sep 11 18:32:34.089071 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.089050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pqb7\" (UniqueName: \"kubernetes.io/projected/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-kube-api-access-4pqb7\") pod \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") "
Sep 11 18:32:34.089134 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.089120    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-maas-api-tls\") pod \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") "
Sep 11 18:32:34.089224 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.089175    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-metrics-certs\") pod \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\" (UID: \"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4\") "
Sep 11 18:32:34.089382 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.089258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "760e5a91-fb45-49d8-a0a6-9b0b0373a2e4" (UID: "760e5a91-fb45-49d8-a0a6-9b0b0373a2e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:32:34.089528 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.089504    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:32:34.091316 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.091279    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "760e5a91-fb45-49d8-a0a6-9b0b0373a2e4" (UID: "760e5a91-fb45-49d8-a0a6-9b0b0373a2e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:32:34.091427 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.091360    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "760e5a91-fb45-49d8-a0a6-9b0b0373a2e4" (UID: "760e5a91-fb45-49d8-a0a6-9b0b0373a2e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:32:34.091427 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.091403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-kube-api-access-4pqb7" (OuterVolumeSpecName: "kube-api-access-4pqb7") pod "760e5a91-fb45-49d8-a0a6-9b0b0373a2e4" (UID: "760e5a91-fb45-49d8-a0a6-9b0b0373a2e4"). InnerVolumeSpecName "kube-api-access-4pqb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:32:34.190755 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.190698    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:32:34.190755 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.190733    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:32:34.190755 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.190751    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pqb7\" (UniqueName: \"kubernetes.io/projected/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4-kube-api-access-4pqb7\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:32:34.305486 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.305377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:34.463325 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.463284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:34.933821 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.933788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2" event={"ID":"760e5a91-fb45-49d8-a0a6-9b0b0373a2e4","Type":"ContainerDied","Data":"429af9e9cd75d585d5d4c2305a5b329aa92456356d6350b1e32fbec8be7eae66"}
Sep 11 18:32:34.934341 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.933803    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"
Sep 11 18:32:34.934341 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.933909    2580 scope.go:117] "RemoveContainer" containerID="aa034a8db9fed4e52841d78e17c39bfc29f8acf09b75a61b909d3d0092cb2c6c"
Sep 11 18:32:34.962226 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.962176    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"]
Sep 11 18:32:34.970755 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:34.970711    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ad8c202-57749754c6-4j6z2"]
Sep 11 18:32:35.341924 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:35.341807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:35.395829 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:35.395787    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="760e5a91-fb45-49d8-a0a6-9b0b0373a2e4" path="/var/lib/kubelet/pods/760e5a91-fb45-49d8-a0a6-9b0b0373a2e4/volumes"
Sep 11 18:32:36.087022 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:36.086982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:36.829860 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:36.829824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:37.630947 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:37.630913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:38.385833 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:38.385772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:40.910811 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:40.910777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:41.011003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:41.010969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:41.176123 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:41.176015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:41.495762 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:41.495652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:41.600592 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:41.600553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:41.706383 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:41.706347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:43.654687 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:43.654655    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:48.095040 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:48.095004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:48.199100 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:48.199054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:48.292678 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:48.292633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:48.479009 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:48.478964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:54.772987 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:54.772937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:54.873697 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:54.873657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:55.778851 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:55.778809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:56.098932 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:56.098836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:57.035801 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:57.035750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:57.584789 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:57.584750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:58.824901 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:58.824861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:59.457627 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:59.457585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:59.556967 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:59.556921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:32:59.878194 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:32:59.878159    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:00.654392 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:00.654328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:01.571730 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:01.571683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:02.304827 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:02.304789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:03.108248 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:03.108156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:03.206448 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:03.206412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:03.454153 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:03.454108    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:04.018351 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.018313    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"]
Sep 11 18:33:04.018925 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.018905    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760e5a91-fb45-49d8-a0a6-9b0b0373a2e4" containerName="maas-api"
Sep 11 18:33:04.019007 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.018927    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="760e5a91-fb45-49d8-a0a6-9b0b0373a2e4" containerName="maas-api"
Sep 11 18:33:04.019064 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.019020    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="760e5a91-fb45-49d8-a0a6-9b0b0373a2e4" containerName="maas-api"
Sep 11 18:33:04.022548 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.022524    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.024806 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.024781    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-869176-serving-cert\""
Sep 11 18:33:04.036497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.036462    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"]
Sep 11 18:33:04.051227 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.051180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:04.071472 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.071434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/875cef81-64fb-4faa-8727-bde9c50e80f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.071472 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.071478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.071713 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.071620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsh45\" (UniqueName: \"kubernetes.io/projected/875cef81-64fb-4faa-8727-bde9c50e80f5-kube-api-access-gsh45\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.071713 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.071678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.151097 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.151057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:04.173141 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.173092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsh45\" (UniqueName: \"kubernetes.io/projected/875cef81-64fb-4faa-8727-bde9c50e80f5-kube-api-access-gsh45\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.173369 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.173151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.173369 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.173228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/875cef81-64fb-4faa-8727-bde9c50e80f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.173369 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.173261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.173369 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:33:04.173315    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-serving-cert: secret "maas-api-e2e-shared-a-w2-869176-serving-cert" not found
Sep 11 18:33:04.173554 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:33:04.173397    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-maas-api-tls podName:875cef81-64fb-4faa-8727-bde9c50e80f5 nodeName:}" failed. No retries permitted until 2026-09-11 18:33:04.673374756 +0000 UTC m=+1065.769968389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-maas-api-tls") pod "maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr" (UID: "875cef81-64fb-4faa-8727-bde9c50e80f5") : secret "maas-api-e2e-shared-a-w2-869176-serving-cert" not found
Sep 11 18:33:04.173895 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.173863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/875cef81-64fb-4faa-8727-bde9c50e80f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.175827 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.175801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.182188 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.182153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsh45\" (UniqueName: \"kubernetes.io/projected/875cef81-64fb-4faa-8727-bde9c50e80f5-kube-api-access-gsh45\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.250948 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.250913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:04.585192 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.585152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:04.678926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.678881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.681424 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.681400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:04.724095 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.724046    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:04.933883 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:04.933832    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:05.210665 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:05.210637    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"]
Sep 11 18:33:05.213100 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:33:05.213070    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875cef81_64fb_4faa_8727_bde9c50e80f5.slice/crio-e42cb4a01ae2bc0584610cbdc6f6881dd382422660918ea23994aa151e6d61d2 WatchSource:0}: Error finding container e42cb4a01ae2bc0584610cbdc6f6881dd382422660918ea23994aa151e6d61d2: Status 404 returned error can't find the container with id e42cb4a01ae2bc0584610cbdc6f6881dd382422660918ea23994aa151e6d61d2
Sep 11 18:33:05.223915 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:05.223868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:06.087439 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:06.087339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr" event={"ID":"875cef81-64fb-4faa-8727-bde9c50e80f5","Type":"ContainerStarted","Data":"0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130"}
Sep 11 18:33:06.087439 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:06.087414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr" event={"ID":"875cef81-64fb-4faa-8727-bde9c50e80f5","Type":"ContainerStarted","Data":"e42cb4a01ae2bc0584610cbdc6f6881dd382422660918ea23994aa151e6d61d2"}
Sep 11 18:33:06.087632 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:06.087471    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:06.097350 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:06.097307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:06.107902 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:06.107845    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr" podStartSLOduration=2.686834611 podStartE2EDuration="3.107826437s" totalImagesPullingTime="420.991826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:06.105487293 +0000 UTC m=+1067.202080919" watchObservedRunningTime="2026-09-11 18:33:06.107826437 +0000 UTC m=+1067.204420064"
Sep 11 18:33:08.724617 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:08.724557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:08.850535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:08.850497    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:09.662453 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:09.662417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:10.393800 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:10.393755    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:11.229666 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:11.229626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:12.097672 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:12.097644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:33:12.244131 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:12.244090    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:13.148707 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:13.148667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:14.052702 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:14.052660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:14.956979 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:14.956942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:15.753434 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:15.753381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:16.668943 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:16.668905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:17.534710 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:17.534667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:18.432066 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:18.432031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:19.195098 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:19.195049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:19.351323 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:19.351281    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:19.716026 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:19.715988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:19.968174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:19.968069    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:20.063684 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:20.063649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:20.527838 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:20.527798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:20.854858 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:20.854727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:21.482015 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.481972    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"]
Sep 11 18:33:21.482305 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.482269    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp" podUID="03105077-3a3f-4b8f-b6d3-7660c7d75f44" containerName="maas-api" containerID="cri-o://4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e" gracePeriod=30
Sep 11 18:33:21.510874 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.510820    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:21.612477 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.612434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:21.713585 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.713548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:21.742931 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.742841    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:33:21.811994 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.811959    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:21.839084 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.839054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03105077-3a3f-4b8f-b6d3-7660c7d75f44-openshift-service-ca\") pod \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") "
Sep 11 18:33:21.839304 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.839102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-metrics-certs\") pod \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") "
Sep 11 18:33:21.839304 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.839133    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txkqv\" (UniqueName: \"kubernetes.io/projected/03105077-3a3f-4b8f-b6d3-7660c7d75f44-kube-api-access-txkqv\") pod \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") "
Sep 11 18:33:21.839304 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.839194    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-maas-api-tls\") pod \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\" (UID: \"03105077-3a3f-4b8f-b6d3-7660c7d75f44\") "
Sep 11 18:33:21.839518 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.839494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03105077-3a3f-4b8f-b6d3-7660c7d75f44-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03105077-3a3f-4b8f-b6d3-7660c7d75f44" (UID: "03105077-3a3f-4b8f-b6d3-7660c7d75f44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:33:21.841471 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.841437    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "03105077-3a3f-4b8f-b6d3-7660c7d75f44" (UID: "03105077-3a3f-4b8f-b6d3-7660c7d75f44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:33:21.841595 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.841486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03105077-3a3f-4b8f-b6d3-7660c7d75f44" (UID: "03105077-3a3f-4b8f-b6d3-7660c7d75f44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:33:21.841595 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.841569    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03105077-3a3f-4b8f-b6d3-7660c7d75f44-kube-api-access-txkqv" (OuterVolumeSpecName: "kube-api-access-txkqv") pod "03105077-3a3f-4b8f-b6d3-7660c7d75f44" (UID: "03105077-3a3f-4b8f-b6d3-7660c7d75f44"). InnerVolumeSpecName "kube-api-access-txkqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:33:21.940302 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.940262    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03105077-3a3f-4b8f-b6d3-7660c7d75f44-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:33:21.940302 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.940300    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:33:21.940302 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.940310    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txkqv\" (UniqueName: \"kubernetes.io/projected/03105077-3a3f-4b8f-b6d3-7660c7d75f44-kube-api-access-txkqv\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:33:21.940765 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:21.940321    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03105077-3a3f-4b8f-b6d3-7660c7d75f44-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:33:22.140584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.140546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:22.170233 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.170167    2580 generic.go:356] "Generic (PLEG): container finished" podID="03105077-3a3f-4b8f-b6d3-7660c7d75f44" containerID="4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e" exitCode=0
Sep 11 18:33:22.170416 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.170249    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"
Sep 11 18:33:22.170416 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.170239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp" event={"ID":"03105077-3a3f-4b8f-b6d3-7660c7d75f44","Type":"ContainerDied","Data":"4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e"}
Sep 11 18:33:22.170416 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.170375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp" event={"ID":"03105077-3a3f-4b8f-b6d3-7660c7d75f44","Type":"ContainerDied","Data":"f3a1798c01f35624f7949c34da4d5860e1cb9b5c19b785e4dc94f1b04c1658c3"}
Sep 11 18:33:22.170416 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.170402    2580 scope.go:117] "RemoveContainer" containerID="4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e"
Sep 11 18:33:22.182644 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.182616    2580 scope.go:117] "RemoveContainer" containerID="4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e"
Sep 11 18:33:22.183027 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:33:22.183003    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e\": container with ID starting with 4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e not found: ID does not exist" containerID="4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e"
Sep 11 18:33:22.183102 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.183044    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e"} err="failed to get container status \"4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e\": rpc error: code = NotFound desc = could not find container \"4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e\": container with ID starting with 4f0594cd92cca0c0b5eaa5da7ae85bbb90660a425fcde090de1b1cc8e8340c5e not found: ID does not exist"
Sep 11 18:33:22.195459 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.195418    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"]
Sep 11 18:33:22.201690 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.201618    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:22.201885 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:22.201711    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c7c83f-7f6b56b5d8-4lzdp"]
Sep 11 18:33:23.142273 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:23.142235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:23.397249 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:23.397146    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03105077-3a3f-4b8f-b6d3-7660c7d75f44" path="/var/lib/kubelet/pods/03105077-3a3f-4b8f-b6d3-7660c7d75f44/volumes"
Sep 11 18:33:23.800428 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:23.800320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:23.901981 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:23.901942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:24.158486 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:24.158450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:24.952739 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:24.952698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:25.839218 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:25.839164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:26.584806 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:26.584761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:27.403818 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.403767    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"]
Sep 11 18:33:27.404927 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.404897    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03105077-3a3f-4b8f-b6d3-7660c7d75f44" containerName="maas-api"
Sep 11 18:33:27.404927 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.404926    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="03105077-3a3f-4b8f-b6d3-7660c7d75f44" containerName="maas-api"
Sep 11 18:33:27.405125 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.405027    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="03105077-3a3f-4b8f-b6d3-7660c7d75f44" containerName="maas-api"
Sep 11 18:33:27.409659 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.409632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.411919 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.411892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-0ead89-serving-cert\""
Sep 11 18:33:27.416619 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.416544    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"]
Sep 11 18:33:27.439237 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.439184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:27.494324 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.494284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r85nk\" (UniqueName: \"kubernetes.io/projected/91916eba-fb73-46cd-ab4a-6554531cbbba-kube-api-access-r85nk\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.494537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.494357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91916eba-fb73-46cd-ab4a-6554531cbbba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.494537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.494423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.494537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.494485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.543241 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.543184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:27.595582 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.595536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r85nk\" (UniqueName: \"kubernetes.io/projected/91916eba-fb73-46cd-ab4a-6554531cbbba-kube-api-access-r85nk\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.595791 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.595612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91916eba-fb73-46cd-ab4a-6554531cbbba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.595791 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.595665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.595791 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.595694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.595973 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:33:27.595780    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-serving-cert: secret "maas-api-e2e-shared-b-w2-0ead89-serving-cert" not found
Sep 11 18:33:27.595973 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:33:27.595862    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-maas-api-tls podName:91916eba-fb73-46cd-ab4a-6554531cbbba nodeName:}" failed. No retries permitted until 2026-09-11 18:33:28.095844344 +0000 UTC m=+1089.192437969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-maas-api-tls") pod "maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk" (UID: "91916eba-fb73-46cd-ab4a-6554531cbbba") : secret "maas-api-e2e-shared-b-w2-0ead89-serving-cert" not found
Sep 11 18:33:27.596354 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.596332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91916eba-fb73-46cd-ab4a-6554531cbbba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.598352 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.598324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.606619 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.606594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r85nk\" (UniqueName: \"kubernetes.io/projected/91916eba-fb73-46cd-ab4a-6554531cbbba-kube-api-access-r85nk\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:27.709737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:27.709639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:28.041101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:28.040993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:28.101820 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:28.101769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:28.104282 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:28.104261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:28.140730 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:28.140687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:28.323627 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:28.323523    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:28.559691 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:28.559643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:28.604093 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:28.603879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"]
Sep 11 18:33:28.607042 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:33:28.607012    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91916eba_fb73_46cd_ab4a_6554531cbbba.slice/crio-c27c36a40942408a8d23d59e5c07d630a7dc04dd12cd6f669188381bd9106446 WatchSource:0}: Error finding container c27c36a40942408a8d23d59e5c07d630a7dc04dd12cd6f669188381bd9106446: Status 404 returned error can't find the container with id c27c36a40942408a8d23d59e5c07d630a7dc04dd12cd6f669188381bd9106446
Sep 11 18:33:29.206057 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:29.206020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk" event={"ID":"91916eba-fb73-46cd-ab4a-6554531cbbba","Type":"ContainerStarted","Data":"c27c36a40942408a8d23d59e5c07d630a7dc04dd12cd6f669188381bd9106446"}
Sep 11 18:33:29.337495 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:29.337456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:29.604971 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:29.604858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:29.672048 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:29.672002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:29.898675 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:29.898628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:30.212626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.212524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk" event={"ID":"91916eba-fb73-46cd-ab4a-6554531cbbba","Type":"ContainerStarted","Data":"24faf8afe8268b0fce04cc8abb56b2dae938cdb769a1b3b1dbcf5035d73c9180"}
Sep 11 18:33:30.212809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.212642    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:30.234116 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.234048    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk" podStartSLOduration=2.525627688 podStartE2EDuration="3.234028602s" totalImagesPullingTime="708.400914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:30.232187632 +0000 UTC m=+1091.328781262" watchObservedRunningTime="2026-09-11 18:33:30.234028602 +0000 UTC m=+1091.330622232"
Sep 11 18:33:30.326590 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.326552    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:30.605493 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.605396    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"]
Sep 11 18:33:30.609382 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.609355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.611525 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.611495    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-100986-openshift-default-dockercfg-6bt4b\""
Sep 11 18:33:30.621731 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.621697    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"]
Sep 11 18:33:30.636881 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.636837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:30.727514 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.727469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-envoy\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.727514 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.727517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c125f2-a4f8-494f-a741-cb7121356890-istiod-ca-cert\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.727797 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.727548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c125f2-a4f8-494f-a741-cb7121356890-istio-podinfo\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.727797 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.727611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-data\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.727797 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.727652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb4fb\" (UniqueName: \"kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-kube-api-access-qb4fb\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.727957 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.727801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-istio-token\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.727957 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.727850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-socket\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.727957 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.727892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-certs\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.728089 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.727981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-credential-socket\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.828886 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.828835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-credential-socket\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.828886 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.828880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-envoy\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829114 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.828900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c125f2-a4f8-494f-a741-cb7121356890-istiod-ca-cert\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829114 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.828922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c125f2-a4f8-494f-a741-cb7121356890-istio-podinfo\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829114 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.828972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-data\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829114 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.828991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb4fb\" (UniqueName: \"kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-kube-api-access-qb4fb\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829114 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.829064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-istio-token\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829114 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.829093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-socket\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829447 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.829129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-certs\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829447 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.829418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-data\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829554 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.829524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-socket\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829668 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.829645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-credential-socket\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829734 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.829683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-certs\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.829734 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.829693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c125f2-a4f8-494f-a741-cb7121356890-istiod-ca-cert\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.831361 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.831330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-envoy\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.831969 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.831946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c125f2-a4f8-494f-a741-cb7121356890-istio-podinfo\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.837929 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.837894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-istio-token\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.838306 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.838281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb4fb\" (UniqueName: \"kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-kube-api-access-qb4fb\") pod \"e2e-noten-100986-openshift-default-6798c6f69b-h52dp\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") " pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:30.921791 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:30.921745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:31.200692 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:31.200663    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"]
Sep 11 18:33:31.203279 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:33:31.203251    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c125f2_a4f8_494f_a741_cb7121356890.slice/crio-e29eb6fff99cb428e43fee09d5217d7087125187e73192dd7e5ccc4f1d158067 WatchSource:0}: Error finding container e29eb6fff99cb428e43fee09d5217d7087125187e73192dd7e5ccc4f1d158067: Status 404 returned error can't find the container with id e29eb6fff99cb428e43fee09d5217d7087125187e73192dd7e5ccc4f1d158067
Sep 11 18:33:31.205594 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:31.205557    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:33:31.205709 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:31.205638    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:33:31.205709 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:31.205667    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:33:31.218377 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:31.218338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp" event={"ID":"b1c125f2-a4f8-494f-a741-cb7121356890","Type":"ContainerStarted","Data":"e29eb6fff99cb428e43fee09d5217d7087125187e73192dd7e5ccc4f1d158067"}
Sep 11 18:33:31.697746 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:31.697708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:32.225096 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:32.225057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp" event={"ID":"b1c125f2-a4f8-494f-a741-cb7121356890","Type":"ContainerStarted","Data":"a2936ffde21e141cbf6a5851172c5f4a5e475283b560dc5d79a98ced8d46f26b"}
Sep 11 18:33:32.247247 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:32.247175    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp" podStartSLOduration=2.247158347 podStartE2EDuration="2.247158347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:32.245151698 +0000 UTC m=+1093.341745330" watchObservedRunningTime="2026-09-11 18:33:32.247158347 +0000 UTC m=+1093.343751977"
Sep 11 18:33:32.542110 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:32.541972    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:32.642295 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:32.642253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:32.746013 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:32.745973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:32.922405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:32.922370    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:32.927966 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:32.927937    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:33.230625 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:33.230525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:33.231603 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:33.231581    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:33:36.224107 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:36.224067    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:33:49.709235 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:49.709172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:50.622620 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:50.622580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:51.364223 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:51.364168    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:52.231746 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:52.231706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:52.938993 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:52.938957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:53.755153 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:53.755115    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:53.857838 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:53.857790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:54.503547 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:54.503510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:33:59.645907 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:33:59.645866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:04.252118 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:04.252071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:04.533167 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:04.533079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:04.709045 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:04.709000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:07.349800 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:07.349761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:07.996261 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:07.996220    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:08.082674 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:08.082635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:08.309895 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:08.309798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:09.135419 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:09.135378    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:09.932751 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:09.932714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:10.756237 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:10.756189    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:11.468446 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:11.468399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:12.333098 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:12.333059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:12.575760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:12.575687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:13.215396 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.215360    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"]
Sep 11 18:34:13.220866 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.220841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.223047 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.223020    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-18a45f6a-serving-cert\""
Sep 11 18:34:13.228077 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.228045    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"]
Sep 11 18:34:13.248072 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.248039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:13.334260 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.334179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-metrics-certs\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.334704 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.334378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8224c6-c763-4d5a-94ad-57411277a1d8-openshift-service-ca\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.334704 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.334439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-maas-api-tls\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.334704 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.334504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7zs\" (UniqueName: \"kubernetes.io/projected/cb8224c6-c763-4d5a-94ad-57411277a1d8-kube-api-access-pj7zs\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.349235 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.349187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:13.435379 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.435345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7zs\" (UniqueName: \"kubernetes.io/projected/cb8224c6-c763-4d5a-94ad-57411277a1d8-kube-api-access-pj7zs\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.435581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.435388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-metrics-certs\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.435581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.435439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8224c6-c763-4d5a-94ad-57411277a1d8-openshift-service-ca\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.435581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.435469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-maas-api-tls\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.435581 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:34:13.435569    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-serving-cert: secret "maas-api-e2e-mt-18a45f6a-serving-cert" not found
Sep 11 18:34:13.435768 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:34:13.435645    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-maas-api-tls podName:cb8224c6-c763-4d5a-94ad-57411277a1d8 nodeName:}" failed. No retries permitted until 2026-09-11 18:34:13.935625878 +0000 UTC m=+1135.032219487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-maas-api-tls") pod "maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n" (UID: "cb8224c6-c763-4d5a-94ad-57411277a1d8") : secret "maas-api-e2e-mt-18a45f6a-serving-cert" not found
Sep 11 18:34:13.436141 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.436118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8224c6-c763-4d5a-94ad-57411277a1d8-openshift-service-ca\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.438166 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.438116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-metrics-certs\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.445835 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.445806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7zs\" (UniqueName: \"kubernetes.io/projected/cb8224c6-c763-4d5a-94ad-57411277a1d8-kube-api-access-pj7zs\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.504829 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.504740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:13.897021 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.896982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:13.941073 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.941029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-maas-api-tls\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:13.943538 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:13.943517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-maas-api-tls\") pod \"maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:14.040448 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:14.040413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:14.133972 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:14.133835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:14.416376 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:14.416187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"]
Sep 11 18:34:14.418829 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:34:14.418801    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8224c6_c763_4d5a_94ad_57411277a1d8.slice/crio-a1b826e1df12913015c98cc1129f3c2d8f7c3160377368f21dc99f515b529248 WatchSource:0}: Error finding container a1b826e1df12913015c98cc1129f3c2d8f7c3160377368f21dc99f515b529248: Status 404 returned error can't find the container with id a1b826e1df12913015c98cc1129f3c2d8f7c3160377368f21dc99f515b529248
Sep 11 18:34:14.431361 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:14.431328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n" event={"ID":"cb8224c6-c763-4d5a-94ad-57411277a1d8","Type":"ContainerStarted","Data":"a1b826e1df12913015c98cc1129f3c2d8f7c3160377368f21dc99f515b529248"}
Sep 11 18:34:15.437989 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:15.437953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n" event={"ID":"cb8224c6-c763-4d5a-94ad-57411277a1d8","Type":"ContainerStarted","Data":"bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2"}
Sep 11 18:34:15.438448 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:15.438091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:15.456849 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:15.456794    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n" podStartSLOduration=1.629119448 podStartE2EDuration="2.456775056s" totalImagesPullingTime="827.655608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:15.454560103 +0000 UTC m=+1136.551153734" watchObservedRunningTime="2026-09-11 18:34:15.456775056 +0000 UTC m=+1136.553368721"
Sep 11 18:34:21.449741 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:21.449709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:34:30.064159 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:30.064116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:33.555429 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:33.555393    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"]
Sep 11 18:34:33.565649 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:33.565615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:35.637230 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:35.637156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:36.527329 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:36.527288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:37.391120 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:37.391076    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:38.006480 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:38.006439    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:38.146408 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:38.146366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:39.050622 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:39.050589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:39.781952 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:39.781920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:40.540958 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:40.540923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:41.537337 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:41.537237    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:48.575825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.575776    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v" podUID="dbf989d4-0e9e-45e0-81b5-43e003cf314e" containerName="main" containerID="cri-o://fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2" gracePeriod=45
Sep 11 18:34:48.826722 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.826647    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:34:48.974228 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-home\") pod \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") "
Sep 11 18:34:48.974402 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tls-certs\") pod \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") "
Sep 11 18:34:48.974402 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-dshm\") pod \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") "
Sep 11 18:34:48.974402 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-model-cache\") pod \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") "
Sep 11 18:34:48.974539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4bwc\" (UniqueName: \"kubernetes.io/projected/dbf989d4-0e9e-45e0-81b5-43e003cf314e-kube-api-access-n4bwc\") pod \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") "
Sep 11 18:34:48.974539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tmp-dir\") pod \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\" (UID: \"dbf989d4-0e9e-45e0-81b5-43e003cf314e\") "
Sep 11 18:34:48.974539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974483    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-home" (OuterVolumeSpecName: "home") pod "dbf989d4-0e9e-45e0-81b5-43e003cf314e" (UID: "dbf989d4-0e9e-45e0-81b5-43e003cf314e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:34:48.974694 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbf989d4-0e9e-45e0-81b5-43e003cf314e" (UID: "dbf989d4-0e9e-45e0-81b5-43e003cf314e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:34:48.974780 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbf989d4-0e9e-45e0-81b5-43e003cf314e" (UID: "dbf989d4-0e9e-45e0-81b5-43e003cf314e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:34:48.974848 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974828    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-home\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:34:48.974848 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.974845    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-model-cache\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:34:48.976652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.976625    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbf989d4-0e9e-45e0-81b5-43e003cf314e" (UID: "dbf989d4-0e9e-45e0-81b5-43e003cf314e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:34:48.976741 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.976654    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-dshm" (OuterVolumeSpecName: "dshm") pod "dbf989d4-0e9e-45e0-81b5-43e003cf314e" (UID: "dbf989d4-0e9e-45e0-81b5-43e003cf314e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:34:48.976798 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:48.976760    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf989d4-0e9e-45e0-81b5-43e003cf314e-kube-api-access-n4bwc" (OuterVolumeSpecName: "kube-api-access-n4bwc") pod "dbf989d4-0e9e-45e0-81b5-43e003cf314e" (UID: "dbf989d4-0e9e-45e0-81b5-43e003cf314e"). InnerVolumeSpecName "kube-api-access-n4bwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:34:49.075975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.075928    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tls-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:34:49.075975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.075966    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-dshm\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:34:49.075975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.075980    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4bwc\" (UniqueName: \"kubernetes.io/projected/dbf989d4-0e9e-45e0-81b5-43e003cf314e-kube-api-access-n4bwc\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:34:49.076266 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.075997    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf989d4-0e9e-45e0-81b5-43e003cf314e-tmp-dir\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:34:49.619577 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.619538    2580 generic.go:356] "Generic (PLEG): container finished" podID="dbf989d4-0e9e-45e0-81b5-43e003cf314e" containerID="fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2" exitCode=0
Sep 11 18:34:49.620003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.619587    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"
Sep 11 18:34:49.620003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.619630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v" event={"ID":"dbf989d4-0e9e-45e0-81b5-43e003cf314e","Type":"ContainerDied","Data":"fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2"}
Sep 11 18:34:49.620003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.619676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v" event={"ID":"dbf989d4-0e9e-45e0-81b5-43e003cf314e","Type":"ContainerDied","Data":"539f9d1967759c951a9b9f44d70f7f6f7b1471cff341c597ee5caf9520279136"}
Sep 11 18:34:49.620003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.619696    2580 scope.go:117] "RemoveContainer" containerID="fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2"
Sep 11 18:34:49.628509 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.628488    2580 scope.go:117] "RemoveContainer" containerID="fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2"
Sep 11 18:34:49.628881 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:34:49.628856    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2\": container with ID starting with fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2 not found: ID does not exist" containerID="fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2"
Sep 11 18:34:49.628922 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.628894    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2"} err="failed to get container status \"fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2\": rpc error: code = NotFound desc = could not find container \"fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2\": container with ID starting with fb54616ffee53c05fb140c7a9dbd19bd0fc48f7d4258b22afb89b8d97e5b6ec2 not found: ID does not exist"
Sep 11 18:34:49.636704 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.636674    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"]
Sep 11 18:34:49.643804 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:49.643763    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7d7509/test-model-7d7509-kserve-8567659f6f-k495v"]
Sep 11 18:34:51.395812 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:51.395767    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbf989d4-0e9e-45e0-81b5-43e003cf314e" path="/var/lib/kubelet/pods/dbf989d4-0e9e-45e0-81b5-43e003cf314e/volumes"
Sep 11 18:34:52.845809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:52.845770    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"]
Sep 11 18:34:52.846229 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:52.846188    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf989d4-0e9e-45e0-81b5-43e003cf314e" containerName="main"
Sep 11 18:34:52.846229 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:52.846213    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf989d4-0e9e-45e0-81b5-43e003cf314e" containerName="main"
Sep 11 18:34:52.846313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:52.846293    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf989d4-0e9e-45e0-81b5-43e003cf314e" containerName="main"
Sep 11 18:34:52.850175 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:52.850150    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:52.852208 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:52.852178    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"e2e-conflict-b-498c8da2-kserve-self-signed-certs\""
Sep 11 18:34:52.856961 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:52.856931    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"]
Sep 11 18:34:52.880377 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:52.880343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:53.015010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.014967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-home\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.015189 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.015015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gg5l\" (UniqueName: \"kubernetes.io/projected/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-kube-api-access-7gg5l\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.015189 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.015125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-model-cache\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.015297 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.015259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tmp-dir\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.015346 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.015301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-dshm\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.015346 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.015331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tls-certs\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.115985 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.115889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-home\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.115985 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.115931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gg5l\" (UniqueName: \"kubernetes.io/projected/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-kube-api-access-7gg5l\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.115985 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.115961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-model-cache\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.116386 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.116010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tmp-dir\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.116386 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.116034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-dshm\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.116386 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.116065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tls-certs\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.116541 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.116445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-home\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.116541 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.116464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-model-cache\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.116541 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.116516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tmp-dir\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.118505 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.118481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-dshm\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.118765 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.118736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tls-certs\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.126414 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.126379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gg5l\" (UniqueName: \"kubernetes.io/projected/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-kube-api-access-7gg5l\") pod \"e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.161149 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.161105    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:34:53.213897 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.213861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:53.441930 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.441902    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"]
Sep 11 18:34:53.444784 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:34:53.444742    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77af7239_2d3f_4a6e_8e2c_c1da8fdd0a6f.slice/crio-a8c355c788e1f4c4159a7b73408a44917d134ee808f1770988ecadfb916b378d WatchSource:0}: Error finding container a8c355c788e1f4c4159a7b73408a44917d134ee808f1770988ecadfb916b378d: Status 404 returned error can't find the container with id a8c355c788e1f4c4159a7b73408a44917d134ee808f1770988ecadfb916b378d
Sep 11 18:34:53.641819 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.641779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9" event={"ID":"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f","Type":"ContainerStarted","Data":"d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f"}
Sep 11 18:34:53.641819 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.641827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9" event={"ID":"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f","Type":"ContainerStarted","Data":"a8c355c788e1f4c4159a7b73408a44917d134ee808f1770988ecadfb916b378d"}
Sep 11 18:34:53.661089 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:53.661026    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9" podStartSLOduration=1.661008209 podStartE2EDuration="1.661008209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:53.659075097 +0000 UTC m=+1174.755668728" watchObservedRunningTime="2026-09-11 18:34:53.661008209 +0000 UTC m=+1174.757601840"
Sep 11 18:34:54.202802 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:54.202747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:55.047730 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:55.047679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:55.807113 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:55.807073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:56.558506 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:56.558465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:57.439915 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:57.439877    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:58.160731 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:58.160692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:59.021270 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:59.021221    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:34:59.758127 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:34:59.758083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:00.277006 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:00.276960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:01.465473 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:01.465429    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:01.551925 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:01.551895    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"]
Sep 11 18:35:01.552135 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:01.552107    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp" podUID="b1c125f2-a4f8-494f-a741-cb7121356890" containerName="istio-proxy" containerID="cri-o://a2936ffde21e141cbf6a5851172c5f4a5e475283b560dc5d79a98ced8d46f26b" gracePeriod=30
Sep 11 18:35:01.578208 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:01.578131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:01.808463 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:01.808362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:01.978185 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:01.978142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:03.161423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.161382    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:35:03.161869 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.161672    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:35:03.174444 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.174413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:35:03.233215 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.233159    2580 patch_prober.go:28] interesting pod/e2e-noten-100986-openshift-default-6798c6f69b-h52dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:35:03.233378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.233247    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp" podUID="b1c125f2-a4f8-494f-a741-cb7121356890" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:35:03.553254 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.553163    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"]
Sep 11 18:35:03.553445 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.553415    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf" podUID="849cdd3b-c6a7-4279-8c1b-2ec478946f0c" containerName="maas-api" containerID="cri-o://3fbe4d10f0b9717e38ba1624994c404ac20366c6ce74322489283737350ce563" gracePeriod=30
Sep 11 18:35:03.626697 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.626658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:03.692146 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.692111    2580 generic.go:356] "Generic (PLEG): container finished" podID="849cdd3b-c6a7-4279-8c1b-2ec478946f0c" containerID="3fbe4d10f0b9717e38ba1624994c404ac20366c6ce74322489283737350ce563" exitCode=0
Sep 11 18:35:03.692347 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.692183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf" event={"ID":"849cdd3b-c6a7-4279-8c1b-2ec478946f0c","Type":"ContainerDied","Data":"3fbe4d10f0b9717e38ba1624994c404ac20366c6ce74322489283737350ce563"}
Sep 11 18:35:03.704537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.704502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:35:03.724144 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.724105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:03.811887 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.811787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:03.825282 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.825250    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:35:03.871721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.871678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:03.919830 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.919796    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-maas-api-tls\") pod \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") "
Sep 11 18:35:03.920031 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.919886    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-openshift-service-ca\") pod \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") "
Sep 11 18:35:03.920031 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.919911    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjmr5\" (UniqueName: \"kubernetes.io/projected/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-kube-api-access-fjmr5\") pod \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") "
Sep 11 18:35:03.920031 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.919971    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-metrics-certs\") pod \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\" (UID: \"849cdd3b-c6a7-4279-8c1b-2ec478946f0c\") "
Sep 11 18:35:03.920360 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.920326    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "849cdd3b-c6a7-4279-8c1b-2ec478946f0c" (UID: "849cdd3b-c6a7-4279-8c1b-2ec478946f0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:03.922144 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.922116    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-kube-api-access-fjmr5" (OuterVolumeSpecName: "kube-api-access-fjmr5") pod "849cdd3b-c6a7-4279-8c1b-2ec478946f0c" (UID: "849cdd3b-c6a7-4279-8c1b-2ec478946f0c"). InnerVolumeSpecName "kube-api-access-fjmr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:03.922315 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.922142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "849cdd3b-c6a7-4279-8c1b-2ec478946f0c" (UID: "849cdd3b-c6a7-4279-8c1b-2ec478946f0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:03.922654 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:03.922628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "849cdd3b-c6a7-4279-8c1b-2ec478946f0c" (UID: "849cdd3b-c6a7-4279-8c1b-2ec478946f0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:04.021663 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.021625    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.021663 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.021658    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.021663 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.021668    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjmr5\" (UniqueName: \"kubernetes.io/projected/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-kube-api-access-fjmr5\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.021663 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.021677    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849cdd3b-c6a7-4279-8c1b-2ec478946f0c-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.142244 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.142208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:04.245216 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.245171    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:04.699012 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.698979    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"
Sep 11 18:35:04.699257 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.699028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf" event={"ID":"849cdd3b-c6a7-4279-8c1b-2ec478946f0c","Type":"ContainerDied","Data":"af5bbc7f5b038207fc0d36e94f8f0fd507b3e05073a0f74c884742d7cc2dc248"}
Sep 11 18:35:04.699257 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.699104    2580 scope.go:117] "RemoveContainer" containerID="3fbe4d10f0b9717e38ba1624994c404ac20366c6ce74322489283737350ce563"
Sep 11 18:35:04.722477 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.722440    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"]
Sep 11 18:35:04.727537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:04.727502    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d7509-8c9c8d77f-tk7nf"]
Sep 11 18:35:05.149356 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:05.149315    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:05.345051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:05.345012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:05.396140 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:05.396100    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="849cdd3b-c6a7-4279-8c1b-2ec478946f0c" path="/var/lib/kubelet/pods/849cdd3b-c6a7-4279-8c1b-2ec478946f0c/volumes"
Sep 11 18:35:05.885625 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:05.885586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:06.711326 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.711290    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1c125f2-a4f8-494f-a741-cb7121356890" containerID="a2936ffde21e141cbf6a5851172c5f4a5e475283b560dc5d79a98ced8d46f26b" exitCode=0
Sep 11 18:35:06.711907 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.711369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp" event={"ID":"b1c125f2-a4f8-494f-a741-cb7121356890","Type":"ContainerDied","Data":"a2936ffde21e141cbf6a5851172c5f4a5e475283b560dc5d79a98ced8d46f26b"}
Sep 11 18:35:06.810802 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.810714    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:35:06.837881 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.837836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:06.849076 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849042    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb4fb\" (UniqueName: \"kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-kube-api-access-qb4fb\") pod \"b1c125f2-a4f8-494f-a741-cb7121356890\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") "
Sep 11 18:35:06.849313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849088    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-certs\") pod \"b1c125f2-a4f8-494f-a741-cb7121356890\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") "
Sep 11 18:35:06.849313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849110    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c125f2-a4f8-494f-a741-cb7121356890-istiod-ca-cert\") pod \"b1c125f2-a4f8-494f-a741-cb7121356890\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") "
Sep 11 18:35:06.849313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-socket\") pod \"b1c125f2-a4f8-494f-a741-cb7121356890\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") "
Sep 11 18:35:06.849313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-envoy\") pod \"b1c125f2-a4f8-494f-a741-cb7121356890\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") "
Sep 11 18:35:06.849551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849322    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-credential-socket\") pod \"b1c125f2-a4f8-494f-a741-cb7121356890\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") "
Sep 11 18:35:06.849551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849356    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-istio-token\") pod \"b1c125f2-a4f8-494f-a741-cb7121356890\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") "
Sep 11 18:35:06.849551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c125f2-a4f8-494f-a741-cb7121356890-istio-podinfo\") pod \"b1c125f2-a4f8-494f-a741-cb7121356890\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") "
Sep 11 18:35:06.849551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-data\") pod \"b1c125f2-a4f8-494f-a741-cb7121356890\" (UID: \"b1c125f2-a4f8-494f-a741-cb7121356890\") "
Sep 11 18:35:06.849753 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849611    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b1c125f2-a4f8-494f-a741-cb7121356890" (UID: "b1c125f2-a4f8-494f-a741-cb7121356890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:06.849753 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849653    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c125f2-a4f8-494f-a741-cb7121356890-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b1c125f2-a4f8-494f-a741-cb7121356890" (UID: "b1c125f2-a4f8-494f-a741-cb7121356890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:06.849753 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b1c125f2-a4f8-494f-a741-cb7121356890" (UID: "b1c125f2-a4f8-494f-a741-cb7121356890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:06.849753 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849701    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b1c125f2-a4f8-494f-a741-cb7121356890" (UID: "b1c125f2-a4f8-494f-a741-cb7121356890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:06.849955 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.849845    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-data" (OuterVolumeSpecName: "istio-data") pod "b1c125f2-a4f8-494f-a741-cb7121356890" (UID: "b1c125f2-a4f8-494f-a741-cb7121356890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:06.850038 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.850013    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:06.850038 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.850038    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c125f2-a4f8-494f-a741-cb7121356890-istiod-ca-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:06.850357 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.850053    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-workload-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:06.850357 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.850066    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-credential-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:06.850357 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.850077    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-data\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:06.851783 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.851721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b1c125f2-a4f8-494f-a741-cb7121356890" (UID: "b1c125f2-a4f8-494f-a741-cb7121356890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:06.851897 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.851874    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-kube-api-access-qb4fb" (OuterVolumeSpecName: "kube-api-access-qb4fb") pod "b1c125f2-a4f8-494f-a741-cb7121356890" (UID: "b1c125f2-a4f8-494f-a741-cb7121356890"). InnerVolumeSpecName "kube-api-access-qb4fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:06.852070 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.852045    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1c125f2-a4f8-494f-a741-cb7121356890-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b1c125f2-a4f8-494f-a741-cb7121356890" (UID: "b1c125f2-a4f8-494f-a741-cb7121356890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:35:06.852132 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.852046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-istio-token" (OuterVolumeSpecName: "istio-token") pod "b1c125f2-a4f8-494f-a741-cb7121356890" (UID: "b1c125f2-a4f8-494f-a741-cb7121356890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:06.950898 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.950859    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c125f2-a4f8-494f-a741-cb7121356890-istio-envoy\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:06.950898 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.950893    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-istio-token\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:06.951119 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.950902    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c125f2-a4f8-494f-a741-cb7121356890-istio-podinfo\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:06.951119 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:06.950935    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb4fb\" (UniqueName: \"kubernetes.io/projected/b1c125f2-a4f8-494f-a741-cb7121356890-kube-api-access-qb4fb\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:07.037682 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:07.037640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:07.718270 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:07.718177    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"
Sep 11 18:35:07.718270 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:07.718186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp" event={"ID":"b1c125f2-a4f8-494f-a741-cb7121356890","Type":"ContainerDied","Data":"e29eb6fff99cb428e43fee09d5217d7087125187e73192dd7e5ccc4f1d158067"}
Sep 11 18:35:07.718270 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:07.718268    2580 scope.go:117] "RemoveContainer" containerID="a2936ffde21e141cbf6a5851172c5f4a5e475283b560dc5d79a98ced8d46f26b"
Sep 11 18:35:07.737011 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:07.736976    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"]
Sep 11 18:35:07.741446 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:07.741412    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-100986-openshift-default-6798c6f69b-h52dp"]
Sep 11 18:35:07.755073 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:07.755032    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:08.494550 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:08.494512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:09.357543 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:09.357498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:09.395447 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:09.395412    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c125f2-a4f8-494f-a741-cb7121356890" path="/var/lib/kubelet/pods/b1c125f2-a4f8-494f-a741-cb7121356890/volumes"
Sep 11 18:35:10.756969 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:10.756925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:10.890553 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:10.890519    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:11.035143 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:11.035042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:12.242523 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:12.242485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:12.994715 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:12.994671    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:13.843906 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:13.843871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:14.754681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:14.754643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:15.526799 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:15.526756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:16.270325 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:16.270288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:17.127935 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:17.127880    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:21.856440 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:21.856401    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"]
Sep 11 18:35:21.856916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:21.856738    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp" podUID="84338234-a45a-4d8e-b181-f4473dadcf99" containerName="maas-api" containerID="cri-o://daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921" gracePeriod=30
Sep 11 18:35:21.891018 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:21.890974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:21.960039 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:21.960007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:22.020744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.020652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:22.112013 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.111930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:22.112255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.112238    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:35:22.168824 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.168768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:22.198772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.198733    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84338234-a45a-4d8e-b181-f4473dadcf99-openshift-service-ca\") pod \"84338234-a45a-4d8e-b181-f4473dadcf99\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") "
Sep 11 18:35:22.198959 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.198799    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-maas-api-tls\") pod \"84338234-a45a-4d8e-b181-f4473dadcf99\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") "
Sep 11 18:35:22.198959 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.198817    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-metrics-certs\") pod \"84338234-a45a-4d8e-b181-f4473dadcf99\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") "
Sep 11 18:35:22.198959 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.198945    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzn27\" (UniqueName: \"kubernetes.io/projected/84338234-a45a-4d8e-b181-f4473dadcf99-kube-api-access-jzn27\") pod \"84338234-a45a-4d8e-b181-f4473dadcf99\" (UID: \"84338234-a45a-4d8e-b181-f4473dadcf99\") "
Sep 11 18:35:22.199255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.199225    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84338234-a45a-4d8e-b181-f4473dadcf99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84338234-a45a-4d8e-b181-f4473dadcf99" (UID: "84338234-a45a-4d8e-b181-f4473dadcf99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:22.201101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.201057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84338234-a45a-4d8e-b181-f4473dadcf99" (UID: "84338234-a45a-4d8e-b181-f4473dadcf99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:22.201254 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.201182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "84338234-a45a-4d8e-b181-f4473dadcf99" (UID: "84338234-a45a-4d8e-b181-f4473dadcf99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:22.201500 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.201471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84338234-a45a-4d8e-b181-f4473dadcf99-kube-api-access-jzn27" (OuterVolumeSpecName: "kube-api-access-jzn27") pod "84338234-a45a-4d8e-b181-f4473dadcf99" (UID: "84338234-a45a-4d8e-b181-f4473dadcf99"). InnerVolumeSpecName "kube-api-access-jzn27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:22.299733 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.299687    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84338234-a45a-4d8e-b181-f4473dadcf99-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:22.299733 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.299720    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:22.299733 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.299730    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84338234-a45a-4d8e-b181-f4473dadcf99-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:22.299733 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.299739    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzn27\" (UniqueName: \"kubernetes.io/projected/84338234-a45a-4d8e-b181-f4473dadcf99-kube-api-access-jzn27\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:35:22.467819 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.467776    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:22.570547 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.570511    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:22.796886 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.796791    2580 generic.go:356] "Generic (PLEG): container finished" podID="84338234-a45a-4d8e-b181-f4473dadcf99" containerID="daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921" exitCode=0
Sep 11 18:35:22.796886 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.796838    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"
Sep 11 18:35:22.796886 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.796854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp" event={"ID":"84338234-a45a-4d8e-b181-f4473dadcf99","Type":"ContainerDied","Data":"daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921"}
Sep 11 18:35:22.797131 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.796896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp" event={"ID":"84338234-a45a-4d8e-b181-f4473dadcf99","Type":"ContainerDied","Data":"62de2b0e703e9b860976c38fe5fffdefb37cf7e2b7e31283632668e64a1c48f6"}
Sep 11 18:35:22.797131 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.796918    2580 scope.go:117] "RemoveContainer" containerID="daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921"
Sep 11 18:35:22.807041 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.807013    2580 scope.go:117] "RemoveContainer" containerID="daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921"
Sep 11 18:35:22.807402 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:35:22.807382    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921\": container with ID starting with daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921 not found: ID does not exist" containerID="daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921"
Sep 11 18:35:22.807468 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.807412    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921"} err="failed to get container status \"daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921\": rpc error: code = NotFound desc = could not find container \"daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921\": container with ID starting with daa9ab1f354cdaf28fdc248317edadc5a9dca07f6587ae6e1d0d65d382e3e921 not found: ID does not exist"
Sep 11 18:35:22.822669 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.822635    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"]
Sep 11 18:35:22.825993 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:22.825962    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1de19d-86984c8988-kl2cp"]
Sep 11 18:35:23.396583 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:23.396547    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84338234-a45a-4d8e-b181-f4473dadcf99" path="/var/lib/kubelet/pods/84338234-a45a-4d8e-b181-f4473dadcf99/volumes"
Sep 11 18:35:23.423795 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:23.423756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:24.177972 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:24.177921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:24.929561 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:24.929518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:25.851003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:25.850966    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:26.644893 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:26.644854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:28.565900 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:28.565860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:28.671760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:28.671722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:28.826709 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:28.826609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:29.360360 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.360321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:29.677325 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.677281    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"]
Sep 11 18:35:29.677773 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.677720    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="849cdd3b-c6a7-4279-8c1b-2ec478946f0c" containerName="maas-api"
Sep 11 18:35:29.677773 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.677738    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="849cdd3b-c6a7-4279-8c1b-2ec478946f0c" containerName="maas-api"
Sep 11 18:35:29.677773 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.677760    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84338234-a45a-4d8e-b181-f4473dadcf99" containerName="maas-api"
Sep 11 18:35:29.677773 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.677769    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84338234-a45a-4d8e-b181-f4473dadcf99" containerName="maas-api"
Sep 11 18:35:29.677960 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.677794    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c125f2-a4f8-494f-a741-cb7121356890" containerName="istio-proxy"
Sep 11 18:35:29.677960 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.677800    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c125f2-a4f8-494f-a741-cb7121356890" containerName="istio-proxy"
Sep 11 18:35:29.677960 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.677862    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84338234-a45a-4d8e-b181-f4473dadcf99" containerName="maas-api"
Sep 11 18:35:29.677960 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.677874    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c125f2-a4f8-494f-a741-cb7121356890" containerName="istio-proxy"
Sep 11 18:35:29.677960 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.677881    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="849cdd3b-c6a7-4279-8c1b-2ec478946f0c" containerName="maas-api"
Sep 11 18:35:29.682268 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.682240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.684716 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.684693    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-7ca6df-openshift-default-dockercfg-lbhbc\""
Sep 11 18:35:29.693262 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.693228    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"]
Sep 11 18:35:29.705740 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.705701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:29.770306 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.770267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-data\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.770306 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.770307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-podinfo\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.770603 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.770334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhrm\" (UniqueName: \"kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-kube-api-access-jfhrm\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.770603 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.770367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istiod-ca-cert\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.770603 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.770422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-credential-socket\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.770603 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.770460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-token\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.770603 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.770561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-envoy\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.770794 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.770607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-socket\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.770794 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.770677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-certs\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-data\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-podinfo\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872428 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhrm\" (UniqueName: \"kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-kube-api-access-jfhrm\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872428 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istiod-ca-cert\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872428 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-credential-socket\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872428 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-token\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872428 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-envoy\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872428 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-socket\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872428 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-certs\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872793 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-credential-socket\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872793 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-socket\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872889 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-certs\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.872889 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.872850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-data\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.873061 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.873042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istiod-ca-cert\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.874580 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.874557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-envoy\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.874854 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.874833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-podinfo\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.882314 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.882278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-token\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.882522 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.882498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhrm\" (UniqueName: \"kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-kube-api-access-jfhrm\") pod \"e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") " pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:29.995741 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:29.995641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:30.283226 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:30.283185    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"]
Sep 11 18:35:30.285663 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:35:30.285627    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbda8b7d_4c3b_498f_bc1a_5995a49c41db.slice/crio-1533c6320331e1b0f767ebe55ae1bdca982c1fd3f42b1cfdd95d562d8824a183 WatchSource:0}: Error finding container 1533c6320331e1b0f767ebe55ae1bdca982c1fd3f42b1cfdd95d562d8824a183: Status 404 returned error can't find the container with id 1533c6320331e1b0f767ebe55ae1bdca982c1fd3f42b1cfdd95d562d8824a183
Sep 11 18:35:30.287993 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:30.287957    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:35:30.288063 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:30.288033    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:35:30.288100 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:30.288063    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:35:30.375032 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:30.374994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:30.838488 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:30.838451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" event={"ID":"bbda8b7d-4c3b-498f-bc1a-5995a49c41db","Type":"ContainerStarted","Data":"aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274"}
Sep 11 18:35:30.838916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:30.838509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" event={"ID":"bbda8b7d-4c3b-498f-bc1a-5995a49c41db","Type":"ContainerStarted","Data":"1533c6320331e1b0f767ebe55ae1bdca982c1fd3f42b1cfdd95d562d8824a183"}
Sep 11 18:35:30.859570 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:30.859510    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" podStartSLOduration=1.85949212 podStartE2EDuration="1.85949212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:30.858375733 +0000 UTC m=+1211.954969366" watchObservedRunningTime="2026-09-11 18:35:30.85949212 +0000 UTC m=+1211.956085750"
Sep 11 18:35:30.996321 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:30.996278    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:31.949085 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:31.949042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:32.001671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:32.001637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:32.028165 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:32.028124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:32.086893 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:32.086859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:32.784734 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:32.784694    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:32.848164 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:32.848124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:32.849331 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:32.849304    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:35:32.935466 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:32.935417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:33.024917 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:33.024870    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:33.126223 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:33.126156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:33.836232 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:33.836178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:34.894999 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:34.894959    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:35.748134 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:35.748092    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:36.467488 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:36.467451    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:37.361872 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:37.361803    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:37.424247 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:37.424173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:37.527695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:37.527663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:37.876555 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:37.876520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:37.976171 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:37.976128    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:38.940970 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:38.940931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:39.825840 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:39.825798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:43.994361 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:43.994316    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:35:48.530476 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:35:48.530427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:00.027956 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:00.027914    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:00.777733 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:00.777686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:01.556296 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:01.556257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:02.356171 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:02.356130    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:02.851256 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:02.851191    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 11 18:36:02.851700 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:02.851284    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 11 18:36:03.009172 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:03.009065    2580 generic.go:356] "Generic (PLEG): container finished" podID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerID="aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274" exitCode=0
Sep 11 18:36:03.009172 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:03.009138    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" event={"ID":"bbda8b7d-4c3b-498f-bc1a-5995a49c41db","Type":"ContainerDied","Data":"aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274"}
Sep 11 18:36:03.009686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:03.009652    2580 scope.go:117] "RemoveContainer" containerID="aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274"
Sep 11 18:36:03.011033 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:03.011001    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:36:03.011105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:03.011074    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:36:03.011147 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:03.011109    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:36:03.144449 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:03.144401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:03.245415 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:03.245372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:03.702677 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:03.702630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:04.016574 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.016468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" event={"ID":"bbda8b7d-4c3b-498f-bc1a-5995a49c41db","Type":"ContainerStarted","Data":"6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8"}
Sep 11 18:36:04.146098 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.146058    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"]
Sep 11 18:36:04.174575 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.174534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"]
Sep 11 18:36:04.175021 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.174926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.178642 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.178614    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-711a34-openshift-default-dockercfg-zgjlh\""
Sep 11 18:36:04.179518 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.179493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:04.272551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.272447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:04.299804 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.299758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-data\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.299804 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.299801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-token\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.300024 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.299818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istiod-ca-cert\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.300024 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.299903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-podinfo\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.300024 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.299971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-socket\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.300024 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.300008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-credential-socket\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.300240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.300111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-envoy\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.300240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.300138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-certs\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.300240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.300158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wqv\" (UniqueName: \"kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-kube-api-access-v6wqv\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.374012 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.373962    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:04.401739 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.401696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-podinfo\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.401739 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.401756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-socket\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402019 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.401789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-credential-socket\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402019 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.401870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-envoy\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402019 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.401894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-certs\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402019 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.401914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wqv\" (UniqueName: \"kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-kube-api-access-v6wqv\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402019 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.401938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-data\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402019 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.401959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-token\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402019 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.401984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istiod-ca-cert\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402444 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.402221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-socket\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402444 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.402279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-certs\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402444 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.402396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-data\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402633 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.402609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-credential-socket\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.402792 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.402770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istiod-ca-cert\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.404323 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.404299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-envoy\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.404448 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.404427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-podinfo\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.410703 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.410675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-token\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.410850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.410810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wqv\" (UniqueName: \"kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-kube-api-access-v6wqv\") pod \"e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") " pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.489470 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.489430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:04.788716 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.788677    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"]
Sep 11 18:36:04.792474 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:36:04.792435    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33e9b58_4ac7_4053_915a_d5f2d8172edc.slice/crio-593b79f20ad94fddc7b3e90559e9e7b1faa581aae8aa3374fca64a0707021ec7 WatchSource:0}: Error finding container 593b79f20ad94fddc7b3e90559e9e7b1faa581aae8aa3374fca64a0707021ec7: Status 404 returned error can't find the container with id 593b79f20ad94fddc7b3e90559e9e7b1faa581aae8aa3374fca64a0707021ec7
Sep 11 18:36:04.795018 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.794982    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:36:04.795083 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.795056    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:36:04.795121 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.795085    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:36:04.996280 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.996235    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:36:04.996479 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.996452    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 11 18:36:04.996545 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:04.996517    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 11 18:36:05.023964 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:05.023926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" event={"ID":"e33e9b58-4ac7-4053-915a-d5f2d8172edc","Type":"ContainerStarted","Data":"c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e"}
Sep 11 18:36:05.024422 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:05.023987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" event={"ID":"e33e9b58-4ac7-4053-915a-d5f2d8172edc","Type":"ContainerStarted","Data":"593b79f20ad94fddc7b3e90559e9e7b1faa581aae8aa3374fca64a0707021ec7"}
Sep 11 18:36:05.042979 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:05.042859    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" podStartSLOduration=1.042841378 podStartE2EDuration="1.042841378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:05.041996373 +0000 UTC m=+1246.138590018" watchObservedRunningTime="2026-09-11 18:36:05.042841378 +0000 UTC m=+1246.139435009"
Sep 11 18:36:05.184803 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:05.184766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:05.489952 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:05.489912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:05.495637 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:05.495608    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:05.635440 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:05.635404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:05.735257 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:05.735193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:05.997097 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:05.997049    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 11 18:36:05.997325 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:05.997117    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 11 18:36:06.029584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:06.029546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:06.030527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:06.030501    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:06.824191 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:06.824146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:06.997082 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:06.997042    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 11 18:36:06.997311 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:06.997103    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 11 18:36:07.405406 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:07.405365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:07.996907 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:07.996866    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 11 18:36:07.997103 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:07.996929    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 11 18:36:08.638039 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:08.638000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:08.763934 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:08.763895    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:08.997127 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:08.997033    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 11 18:36:08.997127 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:08.997099    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 11 18:36:09.534042 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:09.533999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:10.001325 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:10.001291    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:36:10.001735 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:10.001667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:36:10.002525 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:10.002504    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:36:10.271368 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:10.271273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:11.254015 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:11.253919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:12.040036 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:12.039988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:12.992014 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:12.991969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:13.067684 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:13.067626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:13.390061 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:13.390023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:13.531829 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:13.531791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:13.652055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:13.651951    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:31.228654 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:31.228615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:31.328785 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:31.328718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:33.542617 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.542577    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"]
Sep 11 18:36:33.554530 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.554483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.561030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.560992    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"]
Sep 11 18:36:33.568231 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.562677    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"kube-root-ca.crt\""
Sep 11 18:36:33.568231 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.562993    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"openshift-service-ca.crt\""
Sep 11 18:36:33.568231 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.563462    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"default-dockercfg-gn54h\""
Sep 11 18:36:33.568231 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.563653    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"distinct-2-w0-efa8b9-kserve-self-signed-certs\""
Sep 11 18:36:33.584253 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.584220    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:33.603222 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.603161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-home\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.603419 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.603258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-dshm\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.603419 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.603349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tls-certs\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.603419 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.603389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-model-cache\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.603539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.603427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tmp-dir\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.603539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.603495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6gkj\" (UniqueName: \"kubernetes.io/projected/4eb8a648-6c34-40c6-b800-1d9d9e553aea-kube-api-access-p6gkj\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.704953 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.704912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tls-certs\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.705153 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.704968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-model-cache\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.705153 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.704999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tmp-dir\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.705325 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.705147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gkj\" (UniqueName: \"kubernetes.io/projected/4eb8a648-6c34-40c6-b800-1d9d9e553aea-kube-api-access-p6gkj\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.705325 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.705265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-home\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.705325 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.705306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-dshm\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.705488 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.705466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-model-cache\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.705553 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.705493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tmp-dir\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.705615 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.705598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-home\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.707508 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.707486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-dshm\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.707612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.707540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tls-certs\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.713836 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.713810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6gkj\" (UniqueName: \"kubernetes.io/projected/4eb8a648-6c34-40c6-b800-1d9d9e553aea-kube-api-access-p6gkj\") pod \"distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:33.877227 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:33.877170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:34.161512 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:34.161481    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"]
Sep 11 18:36:34.163853 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:36:34.163816    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eb8a648_6c34_40c6_b800_1d9d9e553aea.slice/crio-ea23a11944a7770e7cc2dda009247220c62f7131d9e56555b858a5a8b742ed3e WatchSource:0}: Error finding container ea23a11944a7770e7cc2dda009247220c62f7131d9e56555b858a5a8b742ed3e: Status 404 returned error can't find the container with id ea23a11944a7770e7cc2dda009247220c62f7131d9e56555b858a5a8b742ed3e
Sep 11 18:36:34.193643 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:34.193606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg" event={"ID":"4eb8a648-6c34-40c6-b800-1d9d9e553aea","Type":"ContainerStarted","Data":"ea23a11944a7770e7cc2dda009247220c62f7131d9e56555b858a5a8b742ed3e"}
Sep 11 18:36:35.201588 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:35.201539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg" event={"ID":"4eb8a648-6c34-40c6-b800-1d9d9e553aea","Type":"ContainerStarted","Data":"cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764"}
Sep 11 18:36:35.218822 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:35.218759    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg" podStartSLOduration=2.218733088 podStartE2EDuration="2.218733088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:35.21722009 +0000 UTC m=+1276.313813724" watchObservedRunningTime="2026-09-11 18:36:35.218733088 +0000 UTC m=+1276.315326719"
Sep 11 18:36:35.342259 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:35.342212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:36.147421 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:36.147383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:36.900589 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:36.900538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:36.910213 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:36.910150    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:58604->10.133.0.48:15021: read: connection reset by peer" start-of-body=
Sep 11 18:36:36.910355 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:36.910253    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:58604->10.133.0.48:15021: read: connection reset by peer"
Sep 11 18:36:37.214311 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:37.214189    2580 generic.go:356] "Generic (PLEG): container finished" podID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerID="c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e" exitCode=0
Sep 11 18:36:37.214311 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:37.214264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" event={"ID":"e33e9b58-4ac7-4053-915a-d5f2d8172edc","Type":"ContainerDied","Data":"c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e"}
Sep 11 18:36:37.214714 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:37.214700    2580 scope.go:117] "RemoveContainer" containerID="c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e"
Sep 11 18:36:37.215914 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:37.215881    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:36:37.215973 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:37.215956    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:36:37.216030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:37.216016    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:36:37.670858 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:37.670814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:37.772500 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:37.772456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:38.222030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:38.221992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" event={"ID":"e33e9b58-4ac7-4053-915a-d5f2d8172edc","Type":"ContainerStarted","Data":"4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f"}
Sep 11 18:36:38.489782 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:38.489689    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:38.489982 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:38.489961    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 18:36:38.490044 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:38.490024    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 18:36:38.568330 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:38.568292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:39.363140 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:39.363097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:39.490462 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:39.490422    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 18:36:39.490662 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:39.490483    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 18:36:40.312920 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:40.312876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:40.490125 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:40.490084    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 18:36:40.490564 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:40.490157    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 18:36:41.075762 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:41.075716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:41.490453 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:41.490412    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 18:36:41.490883 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:41.490490    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 18:36:42.490910 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:42.490862    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 18:36:42.491369 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:42.490940    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 18:36:43.490377 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:43.490331    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 18:36:43.490569 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:43.490395    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 18:36:43.878121 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:43.878085    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:43.878504 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:43.878143    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:43.891380 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:43.891346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:44.275444 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:44.275349    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:36:44.493663 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:44.493625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:44.494161 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:44.494127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:44.494929 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:44.494904    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:36:46.744035 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:46.743995    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"]
Sep 11 18:36:46.763029 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:46.762988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:56.043036 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.043002    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"]
Sep 11 18:36:56.046656 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.046628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.049702 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.049669    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"kube-root-ca.crt\""
Sep 11 18:36:56.049702 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.049669    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"openshift-service-ca.crt\""
Sep 11 18:36:56.049946 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.049729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"default-dockercfg-9lxk4\""
Sep 11 18:36:56.049946 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.049697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"auth-test-model-7ca6df-kserve-self-signed-certs\""
Sep 11 18:36:56.054994 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.054882    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"]
Sep 11 18:36:56.064606 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.064567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:36:56.133137 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.133095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55d04b7-206c-48e9-be24-41da4bbc808d-tls-certs\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.133137 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.133146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-model-cache\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.133481 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.133173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx8xk\" (UniqueName: \"kubernetes.io/projected/a55d04b7-206c-48e9-be24-41da4bbc808d-kube-api-access-gx8xk\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.133481 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.133212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-dshm\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.133481 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.133273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-home\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.133481 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.133358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-tmp-dir\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.234072 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.234025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55d04b7-206c-48e9-be24-41da4bbc808d-tls-certs\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.234072 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.234073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-model-cache\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.234353 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.234096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8xk\" (UniqueName: \"kubernetes.io/projected/a55d04b7-206c-48e9-be24-41da4bbc808d-kube-api-access-gx8xk\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.234353 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.234126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-dshm\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.234353 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.234154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-home\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.234353 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.234194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-tmp-dir\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.234554 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.234509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-model-cache\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.234617 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.234566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-home\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.234727 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.234705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-tmp-dir\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.236605 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.236574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-dshm\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.236911 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.236892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55d04b7-206c-48e9-be24-41da4bbc808d-tls-certs\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.243023 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.242996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8xk\" (UniqueName: \"kubernetes.io/projected/a55d04b7-206c-48e9-be24-41da4bbc808d-kube-api-access-gx8xk\") pod \"auth-test-model-7ca6df-kserve-54d48874ff-5697v\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.357723 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.357611    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:36:56.635988 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:56.635946    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"]
Sep 11 18:36:56.638165 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:36:56.638135    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda55d04b7_206c_48e9_be24_41da4bbc808d.slice/crio-bb7dc7acc9b49c6d626cea8bf34cb381ea79d3c32334f4947c060b14c7295a23 WatchSource:0}: Error finding container bb7dc7acc9b49c6d626cea8bf34cb381ea79d3c32334f4947c060b14c7295a23: Status 404 returned error can't find the container with id bb7dc7acc9b49c6d626cea8bf34cb381ea79d3c32334f4947c060b14c7295a23
Sep 11 18:36:57.337129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:57.337090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v" event={"ID":"a55d04b7-206c-48e9-be24-41da4bbc808d","Type":"ContainerStarted","Data":"7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d"}
Sep 11 18:36:57.337129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:57.337135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v" event={"ID":"a55d04b7-206c-48e9-be24-41da4bbc808d","Type":"ContainerStarted","Data":"bb7dc7acc9b49c6d626cea8bf34cb381ea79d3c32334f4947c060b14c7295a23"}
Sep 11 18:36:57.353450 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:57.353389    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v" podStartSLOduration=1.3533676749999999 podStartE2EDuration="1.353367675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:57.352272936 +0000 UTC m=+1298.448866564" watchObservedRunningTime="2026-09-11 18:36:57.353367675 +0000 UTC m=+1298.449961307"
Sep 11 18:36:58.949333 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:36:58.949295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:01.765523 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:01.765472    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9" podUID="77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" containerName="main" containerID="cri-o://d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f" gracePeriod=45
Sep 11 18:37:02.016490 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.016408    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:37:02.205416 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.205381    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gg5l\" (UniqueName: \"kubernetes.io/projected/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-kube-api-access-7gg5l\") pod \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") "
Sep 11 18:37:02.205754 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.205510    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-home\") pod \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") "
Sep 11 18:37:02.205754 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.205558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-dshm\") pod \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") "
Sep 11 18:37:02.205754 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.205595    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-model-cache\") pod \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") "
Sep 11 18:37:02.205754 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.205634    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tmp-dir\") pod \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") "
Sep 11 18:37:02.205754 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.205760    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tls-certs\") pod \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\" (UID: \"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f\") "
Sep 11 18:37:02.206230 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.205885    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-home" (OuterVolumeSpecName: "home") pod "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" (UID: "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:02.206230 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.206055    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" (UID: "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:02.206230 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.206087    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" (UID: "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:02.206404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.206342    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-home\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:37:02.206404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.206365    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-model-cache\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:37:02.206404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.206379    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tmp-dir\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:37:02.208122 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.208097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-dshm" (OuterVolumeSpecName: "dshm") pod "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" (UID: "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:02.208262 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.208235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" (UID: "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:37:02.208367 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.208341    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-kube-api-access-7gg5l" (OuterVolumeSpecName: "kube-api-access-7gg5l") pod "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" (UID: "77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f"). InnerVolumeSpecName "kube-api-access-7gg5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:02.307736 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.307633    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-tls-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:37:02.307736 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.307669    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gg5l\" (UniqueName: \"kubernetes.io/projected/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-kube-api-access-7gg5l\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:37:02.307736 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.307679    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f-dshm\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:37:02.363991 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.363954    2580 generic.go:356] "Generic (PLEG): container finished" podID="77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" containerID="d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f" exitCode=0
Sep 11 18:37:02.364169 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.363996    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"
Sep 11 18:37:02.364169 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.364043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9" event={"ID":"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f","Type":"ContainerDied","Data":"d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f"}
Sep 11 18:37:02.364169 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.364090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9" event={"ID":"77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f","Type":"ContainerDied","Data":"a8c355c788e1f4c4159a7b73408a44917d134ee808f1770988ecadfb916b378d"}
Sep 11 18:37:02.364169 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.364107    2580 scope.go:117] "RemoveContainer" containerID="d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f"
Sep 11 18:37:02.373850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.373828    2580 scope.go:117] "RemoveContainer" containerID="d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f"
Sep 11 18:37:02.374160 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:37:02.374141    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f\": container with ID starting with d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f not found: ID does not exist" containerID="d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f"
Sep 11 18:37:02.374243 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.374171    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f"} err="failed to get container status \"d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f\": rpc error: code = NotFound desc = could not find container \"d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f\": container with ID starting with d6af9425e358d5fa4ab039902d41c45f7bd335c4277fb090d81a26213ff10f2f not found: ID does not exist"
Sep 11 18:37:02.387490 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.387452    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"]
Sep 11 18:37:02.394319 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:02.394276    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-b-498c8da2-kserve-5bb476f7cb-jwvm9"]
Sep 11 18:37:03.396368 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:03.396328    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" path="/var/lib/kubelet/pods/77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f/volumes"
Sep 11 18:37:06.358212 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:06.358158    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:37:06.358627 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:06.358250    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:37:06.371177 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:06.371142    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:37:06.396973 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:06.396933    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:37:16.716030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:16.715982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:17.530158 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:17.530116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:18.548418 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:18.548375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:19.271247 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:19.271172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:19.434878 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:19.434832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:19.969708 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:19.969668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:20.848690 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:20.848634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:21.562457 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:21.562405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:22.120910 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:22.120868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:22.222502 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:22.222448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:22.416851 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:22.416792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:23.234148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:23.234101    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:31.991173 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:31.991136    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"]
Sep 11 18:37:31.991661 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:31.991364    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n" podUID="cb8224c6-c763-4d5a-94ad-57411277a1d8" containerName="maas-api" containerID="cri-o://bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2" gracePeriod=30
Sep 11 18:37:32.021144 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.021086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:32.088928 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.088876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:32.152024 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.151984    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:32.244363 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.244278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:32.254437 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.254408    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:37:32.316958 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.316921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:32.415745 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.415703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8224c6-c763-4d5a-94ad-57411277a1d8-openshift-service-ca\") pod \"cb8224c6-c763-4d5a-94ad-57411277a1d8\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") "
Sep 11 18:37:32.415963 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.415757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-metrics-certs\") pod \"cb8224c6-c763-4d5a-94ad-57411277a1d8\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") "
Sep 11 18:37:32.415963 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.415802    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj7zs\" (UniqueName: \"kubernetes.io/projected/cb8224c6-c763-4d5a-94ad-57411277a1d8-kube-api-access-pj7zs\") pod \"cb8224c6-c763-4d5a-94ad-57411277a1d8\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") "
Sep 11 18:37:32.416089 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.415959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-maas-api-tls\") pod \"cb8224c6-c763-4d5a-94ad-57411277a1d8\" (UID: \"cb8224c6-c763-4d5a-94ad-57411277a1d8\") "
Sep 11 18:37:32.416145 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.416098    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8224c6-c763-4d5a-94ad-57411277a1d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb8224c6-c763-4d5a-94ad-57411277a1d8" (UID: "cb8224c6-c763-4d5a-94ad-57411277a1d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:32.416429 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.416406    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8224c6-c763-4d5a-94ad-57411277a1d8-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:37:32.418810 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.418268    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8224c6-c763-4d5a-94ad-57411277a1d8-kube-api-access-pj7zs" (OuterVolumeSpecName: "kube-api-access-pj7zs") pod "cb8224c6-c763-4d5a-94ad-57411277a1d8" (UID: "cb8224c6-c763-4d5a-94ad-57411277a1d8"). InnerVolumeSpecName "kube-api-access-pj7zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:32.418810 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.418279    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb8224c6-c763-4d5a-94ad-57411277a1d8" (UID: "cb8224c6-c763-4d5a-94ad-57411277a1d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:37:32.418810 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.418743    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cb8224c6-c763-4d5a-94ad-57411277a1d8" (UID: "cb8224c6-c763-4d5a-94ad-57411277a1d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:37:32.517385 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.517292    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:37:32.517385 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.517323    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8224c6-c763-4d5a-94ad-57411277a1d8-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:37:32.517385 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.517333    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj7zs\" (UniqueName: \"kubernetes.io/projected/cb8224c6-c763-4d5a-94ad-57411277a1d8-kube-api-access-pj7zs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:37:32.518371 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.518348    2580 generic.go:356] "Generic (PLEG): container finished" podID="cb8224c6-c763-4d5a-94ad-57411277a1d8" containerID="bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2" exitCode=0
Sep 11 18:37:32.518437 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.518398    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"
Sep 11 18:37:32.518437 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.518423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n" event={"ID":"cb8224c6-c763-4d5a-94ad-57411277a1d8","Type":"ContainerDied","Data":"bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2"}
Sep 11 18:37:32.518501 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.518467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n" event={"ID":"cb8224c6-c763-4d5a-94ad-57411277a1d8","Type":"ContainerDied","Data":"a1b826e1df12913015c98cc1129f3c2d8f7c3160377368f21dc99f515b529248"}
Sep 11 18:37:32.518501 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.518488    2580 scope.go:117] "RemoveContainer" containerID="bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2"
Sep 11 18:37:32.528681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.528653    2580 scope.go:117] "RemoveContainer" containerID="bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2"
Sep 11 18:37:32.528999 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:37:32.528981    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2\": container with ID starting with bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2 not found: ID does not exist" containerID="bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2"
Sep 11 18:37:32.529060 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.529009    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2"} err="failed to get container status \"bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2\": rpc error: code = NotFound desc = could not find container \"bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2\": container with ID starting with bf911ac64b3eab19c20e234651811890f08084744c6590d91b72890a2e2e08e2 not found: ID does not exist"
Sep 11 18:37:32.542780 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.542736    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"]
Sep 11 18:37:32.548438 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.548390    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18a45f6a-8554f5c9f5-mkx5n"]
Sep 11 18:37:32.705606 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:32.705567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:33.396136 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:33.396102    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8224c6-c763-4d5a-94ad-57411277a1d8" path="/var/lib/kubelet/pods/cb8224c6-c763-4d5a-94ad-57411277a1d8/volumes"
Sep 11 18:37:33.441479 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:33.441439    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:34.164532 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:34.164484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:35.041937 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:35.041896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:35.270678 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:35.270645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:35.743975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:35.743937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:36.431840 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:36.431797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:37.272647 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:37.272600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:40.347152 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:40.347103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:42.240827 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:42.240742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:37:59.174560 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:37:59.174514    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:09.666343 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:09.666303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:16.482193 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:16.482157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:17.564690 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:17.564644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:23.842963 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:23.842919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:27.418613 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:27.418568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:38.801664 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:38.801623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:38.902520 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:38.902475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:39.044509 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:39.044462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:40.306147 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:40.306101    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:40.401447 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:40.401384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:40.500970 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:40.500933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:41.085830 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:41.085796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:41.189666 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:41.189625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:41.289865 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:41.289830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:42.510668 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:42.510628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:48.469085 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:48.469047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:48.566501 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:48.566455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:49.048096 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:49.048057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:49.145279 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:49.145242    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:53.945250 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.945215    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"]
Sep 11 18:38:53.945814 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.945794    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" containerName="main"
Sep 11 18:38:53.945910 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.945818    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" containerName="main"
Sep 11 18:38:53.945910 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.945837    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8224c6-c763-4d5a-94ad-57411277a1d8" containerName="maas-api"
Sep 11 18:38:53.945910 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.945847    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8224c6-c763-4d5a-94ad-57411277a1d8" containerName="maas-api"
Sep 11 18:38:53.946062 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.945934    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="77af7239-2d3f-4a6e-8e2c-c1da8fdd0a6f" containerName="main"
Sep 11 18:38:53.946062 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.945952    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8224c6-c763-4d5a-94ad-57411277a1d8" containerName="maas-api"
Sep 11 18:38:53.948948 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.948920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:53.951164 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.951136    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"distinct-2-w4-724327-kserve-self-signed-certs\""
Sep 11 18:38:53.958373 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.958346    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"]
Sep 11 18:38:53.968917 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:53.968882    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:54.118785 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.118739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xsf2\" (UniqueName: \"kubernetes.io/projected/936eb4fd-d5ae-4c12-94df-90884cd7e856-kube-api-access-7xsf2\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.118785 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.118813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-home\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.119060 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.118879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-tmp-dir\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.119060 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.118906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936eb4fd-d5ae-4c12-94df-90884cd7e856-tls-certs\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.119060 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.118965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-dshm\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.119178 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.119071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-model-cache\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.220232 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.220115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xsf2\" (UniqueName: \"kubernetes.io/projected/936eb4fd-d5ae-4c12-94df-90884cd7e856-kube-api-access-7xsf2\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.220232 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.220168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-home\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.220483 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.220251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-tmp-dir\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.220483 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.220288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936eb4fd-d5ae-4c12-94df-90884cd7e856-tls-certs\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.220483 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.220341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-dshm\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.220483 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.220395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-model-cache\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.220692 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.220662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-home\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.220817 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.220701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-tmp-dir\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.220817 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.220715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-model-cache\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.222706 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.222673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-dshm\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.223016 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.222993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936eb4fd-d5ae-4c12-94df-90884cd7e856-tls-certs\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.228391 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.228369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xsf2\" (UniqueName: \"kubernetes.io/projected/936eb4fd-d5ae-4c12-94df-90884cd7e856-kube-api-access-7xsf2\") pod \"distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") " pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.260829 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.260685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:38:54.537685 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.537657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"]
Sep 11 18:38:54.540152 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:38:54.540122    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod936eb4fd_d5ae_4c12_94df_90884cd7e856.slice/crio-34fcff8b93838d166125607333376a32d045004c6e4403e37c2958a2af483385 WatchSource:0}: Error finding container 34fcff8b93838d166125607333376a32d045004c6e4403e37c2958a2af483385: Status 404 returned error can't find the container with id 34fcff8b93838d166125607333376a32d045004c6e4403e37c2958a2af483385
Sep 11 18:38:54.963084 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.963045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r" event={"ID":"936eb4fd-d5ae-4c12-94df-90884cd7e856","Type":"ContainerStarted","Data":"b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4"}
Sep 11 18:38:54.963608 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.963097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r" event={"ID":"936eb4fd-d5ae-4c12-94df-90884cd7e856","Type":"ContainerStarted","Data":"34fcff8b93838d166125607333376a32d045004c6e4403e37c2958a2af483385"}
Sep 11 18:38:54.979846 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:54.979774    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r" podStartSLOduration=1.979757432 podStartE2EDuration="1.979757432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:54.978892708 +0000 UTC m=+1416.075486355" watchObservedRunningTime="2026-09-11 18:38:54.979757432 +0000 UTC m=+1416.076351063"
Sep 11 18:38:55.299513 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:55.299420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:55.358371 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:55.358325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:55.424883 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:55.424838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:38:55.524672 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:38:55.524639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:04.261256 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:04.261188    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:39:04.261256 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:04.261264    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:39:04.279180 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:04.279144    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:39:05.024405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:05.024362    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:39:05.986675 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:05.986629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:06.213977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:06.213936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:24.839686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:24.839649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:27.128536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:27.128492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:33.229369 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:33.229325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:33.294321 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:33.294275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:33.560353 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:33.560245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:33.859224 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:33.859085    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:33.960831 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:33.960791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:35.845664 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:35.845628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:36.497117 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:36.497075    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:36.590160 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:36.590118    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:36.950566 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:36.950530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:37.891979 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:37.891937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:38.639363 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:38.639327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:39.539817 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:39.539783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:40.324016 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:40.323970    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:41.116504 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.116465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:41.808354 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.808305    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"]
Sep 11 18:39:41.812670 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.812639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:41.814959 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.814926    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c1409b-serving-cert\""
Sep 11 18:39:41.820582 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.820552    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"]
Sep 11 18:39:41.842278 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.842243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:41.897882 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.897829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:41.902036 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.901990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edaf227-3348-429c-a1a0-e99873010fcd-openshift-service-ca\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:41.902036 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.902037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-metrics-certs\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:41.902246 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.902115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-maas-api-tls\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:41.902305 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:41.902277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k54g4\" (UniqueName: \"kubernetes.io/projected/9edaf227-3348-429c-a1a0-e99873010fcd-kube-api-access-k54g4\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:42.003548 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.003498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-maas-api-tls\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:42.003774 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.003615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k54g4\" (UniqueName: \"kubernetes.io/projected/9edaf227-3348-429c-a1a0-e99873010fcd-kube-api-access-k54g4\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:42.003774 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:39:42.003648    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-serving-cert: secret "maas-api-e2e-api-a-c1409b-serving-cert" not found
Sep 11 18:39:42.003774 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:39:42.003721    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-maas-api-tls podName:9edaf227-3348-429c-a1a0-e99873010fcd nodeName:}" failed. No retries permitted until 2026-09-11 18:39:42.5037055 +0000 UTC m=+1463.600299109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-maas-api-tls") pod "maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt" (UID: "9edaf227-3348-429c-a1a0-e99873010fcd") : secret "maas-api-e2e-api-a-c1409b-serving-cert" not found
Sep 11 18:39:42.003774 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.003647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edaf227-3348-429c-a1a0-e99873010fcd-openshift-service-ca\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:42.004016 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.003819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-metrics-certs\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:42.004377 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.004355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edaf227-3348-429c-a1a0-e99873010fcd-openshift-service-ca\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:42.006317 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.006295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-metrics-certs\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:42.012134 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.012103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k54g4\" (UniqueName: \"kubernetes.io/projected/9edaf227-3348-429c-a1a0-e99873010fcd-kube-api-access-k54g4\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:42.329409 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.329368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:42.425786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.425747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:42.496651 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.496617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:39:42.508517 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.508478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-maas-api-tls\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:42.511078 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.511052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-maas-api-tls\") pod \"maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:42.727521 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:42.727468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:43.003088 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:43.003061    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"]
Sep 11 18:39:43.004904 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:39:43.004875    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edaf227_3348_429c_a1a0_e99873010fcd.slice/crio-a8e3c5ddb5ed2804e40c56ade7c0d4c24838d36ba3ebae46a1f0075d666a02f2 WatchSource:0}: Error finding container a8e3c5ddb5ed2804e40c56ade7c0d4c24838d36ba3ebae46a1f0075d666a02f2: Status 404 returned error can't find the container with id a8e3c5ddb5ed2804e40c56ade7c0d4c24838d36ba3ebae46a1f0075d666a02f2
Sep 11 18:39:43.006497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:43.006474    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:39:43.214258 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:43.214223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt" event={"ID":"9edaf227-3348-429c-a1a0-e99873010fcd","Type":"ContainerStarted","Data":"a8e3c5ddb5ed2804e40c56ade7c0d4c24838d36ba3ebae46a1f0075d666a02f2"}
Sep 11 18:39:44.221653 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:44.221595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt" event={"ID":"9edaf227-3348-429c-a1a0-e99873010fcd","Type":"ContainerStarted","Data":"82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319"}
Sep 11 18:39:44.222079 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:44.221783    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:39:44.239975 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:44.239913    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt" podStartSLOduration=2.844554433 podStartE2EDuration="3.239895401s" totalImagesPullingTime="395.340968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:44.23898672 +0000 UTC m=+1465.335580346" watchObservedRunningTime="2026-09-11 18:39:44.239895401 +0000 UTC m=+1465.336489032"
Sep 11 18:39:50.232081 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:39:50.232047    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:40:04.983106 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:04.983062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:05.695291 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:05.695245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:06.574262 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:06.574222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:06.675720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:06.675678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:07.020873 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.020834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:07.315539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.315440    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"]
Sep 11 18:40:07.319810 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.319782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.322155 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.322130    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ea789c-openshift-default-dockercfg-7qrmf\""
Sep 11 18:40:07.332809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.332774    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"]
Sep 11 18:40:07.352855 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.352808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:07.461883 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.461832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmrl7\" (UniqueName: \"kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-kube-api-access-rmrl7\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.462093 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.461982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-token\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.462093 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.462076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-podinfo\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.462232 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.462114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-certs\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.462232 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.462157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-socket\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.462343 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.462237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-envoy\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.462343 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.462269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-data\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.462466 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.462341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-credential-socket\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.462466 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.462393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33acb3bf-828a-4380-9643-b0d3db926b3b-istiod-ca-cert\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563237 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-token\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563451 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-podinfo\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563451 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-certs\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563451 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-socket\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563451 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-envoy\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563451 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-data\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563739 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-credential-socket\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563739 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33acb3bf-828a-4380-9643-b0d3db926b3b-istiod-ca-cert\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563739 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmrl7\" (UniqueName: \"kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-kube-api-access-rmrl7\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563907 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-data\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563907 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-certs\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.563907 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.563889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-socket\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.564155 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.564135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-credential-socket\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.564330 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.564303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33acb3bf-828a-4380-9643-b0d3db926b3b-istiod-ca-cert\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.565809 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.565738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-envoy\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.565984 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.565965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-podinfo\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.574949 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.574911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-token\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.575382 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.575151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmrl7\" (UniqueName: \"kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-kube-api-access-rmrl7\") pod \"e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") " pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.592612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.592564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:07.634155 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.634113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:07.753034 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.752981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:07.921516 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.921490    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"]
Sep 11 18:40:07.924332 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:40:07.924283    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33acb3bf_828a_4380_9643_b0d3db926b3b.slice/crio-e10f610b51639318c911d2e7f70d4e523ccc0a116093d4bb96cf35a8d4befc96 WatchSource:0}: Error finding container e10f610b51639318c911d2e7f70d4e523ccc0a116093d4bb96cf35a8d4befc96: Status 404 returned error can't find the container with id e10f610b51639318c911d2e7f70d4e523ccc0a116093d4bb96cf35a8d4befc96
Sep 11 18:40:07.926735 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.926694    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:40:07.926849 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.926775    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:40:07.926849 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:07.926826    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:40:08.332557 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:08.332458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:08.346420 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:08.346380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn" event={"ID":"33acb3bf-828a-4380-9643-b0d3db926b3b","Type":"ContainerStarted","Data":"163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073"}
Sep 11 18:40:08.346607 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:08.346447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn" event={"ID":"33acb3bf-828a-4380-9643-b0d3db926b3b","Type":"ContainerStarted","Data":"e10f610b51639318c911d2e7f70d4e523ccc0a116093d4bb96cf35a8d4befc96"}
Sep 11 18:40:08.368986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:08.368927    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn" podStartSLOduration=1.3689093030000001 podStartE2EDuration="1.368909303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:40:08.365966685 +0000 UTC m=+1489.462560324" watchObservedRunningTime="2026-09-11 18:40:08.368909303 +0000 UTC m=+1489.465502934"
Sep 11 18:40:08.634794 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:08.634730    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:08.642527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:08.642490    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:09.350842 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:09.350804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:09.352049 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:09.352031    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:09.353032 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:09.353013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:10.236635 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:10.236603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:10.338300 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:10.338264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:10.437234 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:10.437181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:11.149024 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:11.148986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:12.250541 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:12.250502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:12.353528 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:12.353490    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:12.836333 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:12.836295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:12.935105 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:12.935062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:13.018026 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:13.017955    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:13.265686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:13.265651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:14.149786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:14.149747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:15.033109 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:15.033073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:15.750899 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:15.750856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:16.642538 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:16.642498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:17.417869 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:17.417829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:18.173511 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:18.173471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:34.579694 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:34.579640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:35.206054 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:35.206016    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:35.588374 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:35.588279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:36.568992 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:36.568954    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:37.463410 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:37.463369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:38.566111 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:38.566060    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:39.409334 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:39.409295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:40.156033 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:40.155994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:40.393394 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:40.393352    2580 patch_prober.go:28] interesting pod/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 18:40:40.393579 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:40.393434    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn" podUID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 18:40:41.037565 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:41.037523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:41.941784 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:41.941741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:42.535012 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:42.534914    2580 generic.go:356] "Generic (PLEG): container finished" podID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerID="163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073" exitCode=0
Sep 11 18:40:42.535012 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:42.534988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn" event={"ID":"33acb3bf-828a-4380-9643-b0d3db926b3b","Type":"ContainerDied","Data":"163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073"}
Sep 11 18:40:42.535431 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:42.535416    2580 scope.go:117] "RemoveContainer" containerID="163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073"
Sep 11 18:40:42.536697 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:42.536660    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:40:42.536819 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:42.536764    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:40:42.536889 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:42.536817    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:40:42.744750 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:42.744710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:43.474782 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:43.474736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:43.544610 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:43.544577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn" event={"ID":"33acb3bf-828a-4380-9643-b0d3db926b3b","Type":"ContainerStarted","Data":"13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744"}
Sep 11 18:40:43.634508 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:43.634464    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:43.634703 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:43.634661    2580 patch_prober.go:28] interesting pod/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 11 18:40:43.634781 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:43.634713    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn" podUID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 11 18:40:44.492072 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:44.492027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:44.639535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:44.639500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:45.376368 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:45.376323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:45.558000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:45.557970    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:45.559083 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:45.559058    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:40:48.255101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:48.255034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:54.002363 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.002311    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"]
Sep 11 18:40:54.002866 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.002601    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt" podUID="9edaf227-3348-429c-a1a0-e99873010fcd" containerName="maas-api" containerID="cri-o://82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319" gracePeriod=30
Sep 11 18:40:54.259600 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.259519    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:40:54.339406 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.339371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-maas-api-tls\") pod \"9edaf227-3348-429c-a1a0-e99873010fcd\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") "
Sep 11 18:40:54.339609 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.339439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-metrics-certs\") pod \"9edaf227-3348-429c-a1a0-e99873010fcd\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") "
Sep 11 18:40:54.339609 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.339457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k54g4\" (UniqueName: \"kubernetes.io/projected/9edaf227-3348-429c-a1a0-e99873010fcd-kube-api-access-k54g4\") pod \"9edaf227-3348-429c-a1a0-e99873010fcd\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") "
Sep 11 18:40:54.339609 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.339478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edaf227-3348-429c-a1a0-e99873010fcd-openshift-service-ca\") pod \"9edaf227-3348-429c-a1a0-e99873010fcd\" (UID: \"9edaf227-3348-429c-a1a0-e99873010fcd\") "
Sep 11 18:40:54.339962 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.339931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edaf227-3348-429c-a1a0-e99873010fcd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9edaf227-3348-429c-a1a0-e99873010fcd" (UID: "9edaf227-3348-429c-a1a0-e99873010fcd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:40:54.341771 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.341734    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9edaf227-3348-429c-a1a0-e99873010fcd" (UID: "9edaf227-3348-429c-a1a0-e99873010fcd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:40:54.341771 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.341743    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9edaf227-3348-429c-a1a0-e99873010fcd" (UID: "9edaf227-3348-429c-a1a0-e99873010fcd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:40:54.341917 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.341802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edaf227-3348-429c-a1a0-e99873010fcd-kube-api-access-k54g4" (OuterVolumeSpecName: "kube-api-access-k54g4") pod "9edaf227-3348-429c-a1a0-e99873010fcd" (UID: "9edaf227-3348-429c-a1a0-e99873010fcd"). InnerVolumeSpecName "kube-api-access-k54g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:40:54.441472 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.441437    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:40:54.441472 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.441471    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edaf227-3348-429c-a1a0-e99873010fcd-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:40:54.441690 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.441481    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k54g4\" (UniqueName: \"kubernetes.io/projected/9edaf227-3348-429c-a1a0-e99873010fcd-kube-api-access-k54g4\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:40:54.441690 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.441491    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edaf227-3348-429c-a1a0-e99873010fcd-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:40:54.604808 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.604720    2580 generic.go:356] "Generic (PLEG): container finished" podID="9edaf227-3348-429c-a1a0-e99873010fcd" containerID="82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319" exitCode=0
Sep 11 18:40:54.604808 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.604768    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"
Sep 11 18:40:54.604808 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.604801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt" event={"ID":"9edaf227-3348-429c-a1a0-e99873010fcd","Type":"ContainerDied","Data":"82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319"}
Sep 11 18:40:54.605076 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.604856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt" event={"ID":"9edaf227-3348-429c-a1a0-e99873010fcd","Type":"ContainerDied","Data":"a8e3c5ddb5ed2804e40c56ade7c0d4c24838d36ba3ebae46a1f0075d666a02f2"}
Sep 11 18:40:54.605076 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.604883    2580 scope.go:117] "RemoveContainer" containerID="82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319"
Sep 11 18:40:54.608166 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.608135    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:54.615181 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.615160    2580 scope.go:117] "RemoveContainer" containerID="82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319"
Sep 11 18:40:54.615543 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:40:54.615516    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319\": container with ID starting with 82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319 not found: ID does not exist" containerID="82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319"
Sep 11 18:40:54.615664 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.615547    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319"} err="failed to get container status \"82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319\": rpc error: code = NotFound desc = could not find container \"82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319\": container with ID starting with 82182f543f21238eadd161571583ac338825afc9d2b0c076582e3697e199c319 not found: ID does not exist"
Sep 11 18:40:54.633807 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.633739    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"]
Sep 11 18:40:54.635769 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.635734    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1409b-dfcddc8bf-w99dt"]
Sep 11 18:40:54.707519 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:54.707488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:55.396937 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:55.396899    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9edaf227-3348-429c-a1a0-e99873010fcd" path="/var/lib/kubelet/pods/9edaf227-3348-429c-a1a0-e99873010fcd/volumes"
Sep 11 18:40:55.517062 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:55.517021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:56.250891 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:56.250820    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:57.320748 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:57.320687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:58.218674 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:58.218638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:58.995000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:58.994962    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:40:59.850150 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:40:59.850111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:00.623868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:00.623829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:00.978556 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:00.978457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:01.079908 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:01.079866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:01.231014 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:01.230917    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:01.385827 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:01.385796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:02.443492 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:02.443448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:02.866059 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:02.865963    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:03.169498 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:03.169447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:04.028511 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:04.028471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:04.738876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:04.738836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:05.439741 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:05.439705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:06.282262 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:06.282223    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:07.131351 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:07.131309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:07.947407 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:07.947366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:11.730142 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:11.730100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:17.364051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:17.364013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:17.468477 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:17.468398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:17.564574 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:17.564534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:17.623253 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:17.623187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:17.942440 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:17.942338    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:18.059251 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:18.059211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:18.953010 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:18.952974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:19.787184 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:19.787079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:20.768529 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:20.768489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:21.551475 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:21.551406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:22.551242 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:22.551182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:23.340863 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:23.340808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:24.133435 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:24.133384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:24.192282 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:24.192235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:24.361334 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:24.361293    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"]
Sep 11 18:41:24.361693 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:24.361555    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn" podUID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerName="istio-proxy" containerID="cri-o://13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744" gracePeriod=30
Sep 11 18:41:24.392295 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:24.388836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:25.050431 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:25.050360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:25.943472 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:25.943422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:26.849176 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:26.848972    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:27.420426 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.420394    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:41:27.491763 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.491709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmrl7\" (UniqueName: \"kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-kube-api-access-rmrl7\") pod \"33acb3bf-828a-4380-9643-b0d3db926b3b\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") "
Sep 11 18:41:27.491763 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.491783    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-envoy\") pod \"33acb3bf-828a-4380-9643-b0d3db926b3b\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") "
Sep 11 18:41:27.491989 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.491823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-podinfo\") pod \"33acb3bf-828a-4380-9643-b0d3db926b3b\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") "
Sep 11 18:41:27.491989 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.491893    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-certs\") pod \"33acb3bf-828a-4380-9643-b0d3db926b3b\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") "
Sep 11 18:41:27.491989 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.491916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-socket\") pod \"33acb3bf-828a-4380-9643-b0d3db926b3b\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") "
Sep 11 18:41:27.491989 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.491951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33acb3bf-828a-4380-9643-b0d3db926b3b-istiod-ca-cert\") pod \"33acb3bf-828a-4380-9643-b0d3db926b3b\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") "
Sep 11 18:41:27.491989 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.491978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-data\") pod \"33acb3bf-828a-4380-9643-b0d3db926b3b\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") "
Sep 11 18:41:27.492327 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492027    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-credential-socket\") pod \"33acb3bf-828a-4380-9643-b0d3db926b3b\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") "
Sep 11 18:41:27.492327 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492118    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-token\") pod \"33acb3bf-828a-4380-9643-b0d3db926b3b\" (UID: \"33acb3bf-828a-4380-9643-b0d3db926b3b\") "
Sep 11 18:41:27.492327 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "33acb3bf-828a-4380-9643-b0d3db926b3b" (UID: "33acb3bf-828a-4380-9643-b0d3db926b3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:27.492492 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492454    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "33acb3bf-828a-4380-9643-b0d3db926b3b" (UID: "33acb3bf-828a-4380-9643-b0d3db926b3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:27.492555 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "33acb3bf-828a-4380-9643-b0d3db926b3b" (UID: "33acb3bf-828a-4380-9643-b0d3db926b3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:27.492610 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492559    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33acb3bf-828a-4380-9643-b0d3db926b3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "33acb3bf-828a-4380-9643-b0d3db926b3b" (UID: "33acb3bf-828a-4380-9643-b0d3db926b3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:27.492658 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "33acb3bf-828a-4380-9643-b0d3db926b3b" (UID: "33acb3bf-828a-4380-9643-b0d3db926b3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:27.492850 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492831    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:27.492922 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492855    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-workload-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:27.492922 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492871    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33acb3bf-828a-4380-9643-b0d3db926b3b-istiod-ca-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:27.492922 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492884    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-data\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:27.492922 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.492897    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-credential-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:27.494456 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.494423    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-kube-api-access-rmrl7" (OuterVolumeSpecName: "kube-api-access-rmrl7") pod "33acb3bf-828a-4380-9643-b0d3db926b3b" (UID: "33acb3bf-828a-4380-9643-b0d3db926b3b"). InnerVolumeSpecName "kube-api-access-rmrl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:27.494589 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.494530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "33acb3bf-828a-4380-9643-b0d3db926b3b" (UID: "33acb3bf-828a-4380-9643-b0d3db926b3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:27.494651 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.494591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "33acb3bf-828a-4380-9643-b0d3db926b3b" (UID: "33acb3bf-828a-4380-9643-b0d3db926b3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:27.494651 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.494602    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "33acb3bf-828a-4380-9643-b0d3db926b3b" (UID: "33acb3bf-828a-4380-9643-b0d3db926b3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:41:27.594323 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.594220    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-token\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:27.594323 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.594265    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmrl7\" (UniqueName: \"kubernetes.io/projected/33acb3bf-828a-4380-9643-b0d3db926b3b-kube-api-access-rmrl7\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:27.594323 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.594278    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-envoy\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:27.594323 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.594287    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33acb3bf-828a-4380-9643-b0d3db926b3b-istio-podinfo\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:27.761411 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.761372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:27.791652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.791613    2580 generic.go:356] "Generic (PLEG): container finished" podID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerID="13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744" exitCode=0
Sep 11 18:41:27.791836 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.791657    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"
Sep 11 18:41:27.791836 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.791696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn" event={"ID":"33acb3bf-828a-4380-9643-b0d3db926b3b","Type":"ContainerDied","Data":"13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744"}
Sep 11 18:41:27.791836 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.791749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn" event={"ID":"33acb3bf-828a-4380-9643-b0d3db926b3b","Type":"ContainerDied","Data":"e10f610b51639318c911d2e7f70d4e523ccc0a116093d4bb96cf35a8d4befc96"}
Sep 11 18:41:27.791836 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.791772    2580 scope.go:117] "RemoveContainer" containerID="13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744"
Sep 11 18:41:27.802152 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.802132    2580 scope.go:117] "RemoveContainer" containerID="163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073"
Sep 11 18:41:27.811423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.811398    2580 scope.go:117] "RemoveContainer" containerID="13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744"
Sep 11 18:41:27.811755 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:41:27.811733    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744\": container with ID starting with 13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744 not found: ID does not exist" containerID="13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744"
Sep 11 18:41:27.811803 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.811767    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744"} err="failed to get container status \"13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744\": rpc error: code = NotFound desc = could not find container \"13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744\": container with ID starting with 13ab54c78c9008882590297bd91727fff7a4da21e1f32c2b7327ecc830beb744 not found: ID does not exist"
Sep 11 18:41:27.811803 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.811786    2580 scope.go:117] "RemoveContainer" containerID="163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073"
Sep 11 18:41:27.812114 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:41:27.812091    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073\": container with ID starting with 163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073 not found: ID does not exist" containerID="163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073"
Sep 11 18:41:27.812257 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.812119    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073"} err="failed to get container status \"163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073\": rpc error: code = NotFound desc = could not find container \"163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073\": container with ID starting with 163dab2f05f573206e1f071d3865627c7e9c78180ed3e9117fae8d4dc6e0b073 not found: ID does not exist"
Sep 11 18:41:27.815460 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.815427    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"]
Sep 11 18:41:27.819681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:27.819648    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea789c-openshift-default-7cb684894d-6mpnn"]
Sep 11 18:41:28.636056 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:28.636014    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:29.396214 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:29.396159    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33acb3bf-828a-4380-9643-b0d3db926b3b" path="/var/lib/kubelet/pods/33acb3bf-828a-4380-9643-b0d3db926b3b/volumes"
Sep 11 18:41:29.438611 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:29.438570    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:30.223292 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:30.223249    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:30.938695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:30.938649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:31.773709 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:31.773667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:32.236499 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:32.236446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:33.147510 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:33.147471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:33.870958 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:33.870919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:34.602065 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:34.602026    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:35.217405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.217359    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"]
Sep 11 18:41:35.217652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.217570    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr" podUID="875cef81-64fb-4faa-8727-bde9c50e80f5" containerName="maas-api" containerID="cri-o://0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130" gracePeriod=30
Sep 11 18:41:35.241752 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.241694    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:35.340293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.340259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:35.439868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.439820    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:35.501679 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.501642    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:41:35.525153 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.525108    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:35.580219 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.580172    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/875cef81-64fb-4faa-8727-bde9c50e80f5-openshift-service-ca\") pod \"875cef81-64fb-4faa-8727-bde9c50e80f5\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") "
Sep 11 18:41:35.580412 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.580269    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-maas-api-tls\") pod \"875cef81-64fb-4faa-8727-bde9c50e80f5\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") "
Sep 11 18:41:35.580412 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.580306    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-metrics-certs\") pod \"875cef81-64fb-4faa-8727-bde9c50e80f5\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") "
Sep 11 18:41:35.580527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.580436    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsh45\" (UniqueName: \"kubernetes.io/projected/875cef81-64fb-4faa-8727-bde9c50e80f5-kube-api-access-gsh45\") pod \"875cef81-64fb-4faa-8727-bde9c50e80f5\" (UID: \"875cef81-64fb-4faa-8727-bde9c50e80f5\") "
Sep 11 18:41:35.580606 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.580577    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875cef81-64fb-4faa-8727-bde9c50e80f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "875cef81-64fb-4faa-8727-bde9c50e80f5" (UID: "875cef81-64fb-4faa-8727-bde9c50e80f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:35.580782 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.580761    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/875cef81-64fb-4faa-8727-bde9c50e80f5-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:35.582624 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.582589    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "875cef81-64fb-4faa-8727-bde9c50e80f5" (UID: "875cef81-64fb-4faa-8727-bde9c50e80f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:35.582761 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.582653    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "875cef81-64fb-4faa-8727-bde9c50e80f5" (UID: "875cef81-64fb-4faa-8727-bde9c50e80f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:35.582815 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.582758    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875cef81-64fb-4faa-8727-bde9c50e80f5-kube-api-access-gsh45" (OuterVolumeSpecName: "kube-api-access-gsh45") pod "875cef81-64fb-4faa-8727-bde9c50e80f5" (UID: "875cef81-64fb-4faa-8727-bde9c50e80f5"). InnerVolumeSpecName "kube-api-access-gsh45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:35.614285 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.614244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:35.682314 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.682278    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsh45\" (UniqueName: \"kubernetes.io/projected/875cef81-64fb-4faa-8727-bde9c50e80f5-kube-api-access-gsh45\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:35.682314 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.682312    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:35.682314 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.682324    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875cef81-64fb-4faa-8727-bde9c50e80f5-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:35.736664 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.736567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:35.835026 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.834986    2580 generic.go:356] "Generic (PLEG): container finished" podID="875cef81-64fb-4faa-8727-bde9c50e80f5" containerID="0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130" exitCode=0
Sep 11 18:41:35.835259 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.835055    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"
Sep 11 18:41:35.835259 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.835082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr" event={"ID":"875cef81-64fb-4faa-8727-bde9c50e80f5","Type":"ContainerDied","Data":"0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130"}
Sep 11 18:41:35.835259 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.835146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr" event={"ID":"875cef81-64fb-4faa-8727-bde9c50e80f5","Type":"ContainerDied","Data":"e42cb4a01ae2bc0584610cbdc6f6881dd382422660918ea23994aa151e6d61d2"}
Sep 11 18:41:35.835259 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.835176    2580 scope.go:117] "RemoveContainer" containerID="0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130"
Sep 11 18:41:35.845800 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.845770    2580 scope.go:117] "RemoveContainer" containerID="0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130"
Sep 11 18:41:35.846232 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:41:35.846183    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130\": container with ID starting with 0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130 not found: ID does not exist" containerID="0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130"
Sep 11 18:41:35.846310 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.846243    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130"} err="failed to get container status \"0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130\": rpc error: code = NotFound desc = could not find container \"0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130\": container with ID starting with 0ca86027f853e4bcdf1961b636b93128883a7c42cb2cbaba7d4ad55637b7c130 not found: ID does not exist"
Sep 11 18:41:35.859552 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.859515    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"]
Sep 11 18:41:35.861472 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.861446    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-869176-5d779b7c4b-pjwsr"]
Sep 11 18:41:35.959717 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:35.959676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:36.729825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:36.729783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:36.833129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:36.833089    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"]
Sep 11 18:41:36.926715 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:36.926678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:37.166429 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:37.166391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:37.397437 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:37.397394    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="875cef81-64fb-4faa-8727-bde9c50e80f5" path="/var/lib/kubelet/pods/875cef81-64fb-4faa-8727-bde9c50e80f5/volumes"
Sep 11 18:41:37.551280 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:37.551177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:38.350791 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:38.350753    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:39.339009 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:39.338969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:40.139998 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:40.139933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:41.045661 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:41.045604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:42.096744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:42.096707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:43.132192 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:43.132155    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:43.231746 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:43.231689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:43.380564 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:43.380522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:43.854137 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:43.854088    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:44.625448 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:44.625405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:45.229001 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.228964    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"]
Sep 11 18:41:45.229256 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.229224    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97" podUID="3652753b-1774-45b0-b34b-6b06eb31a678" containerName="maas-api" containerID="cri-o://00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6" gracePeriod=30
Sep 11 18:41:45.252781 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.252746    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:45.323760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.323723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:45.423942 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.423733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:45.487616 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.487530    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:41:45.505112 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.505073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:45.568091 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.568050    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:45.584092 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.584051    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3652753b-1774-45b0-b34b-6b06eb31a678-openshift-service-ca\") pod \"3652753b-1774-45b0-b34b-6b06eb31a678\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") "
Sep 11 18:41:45.584295 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.584119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-metrics-certs\") pod \"3652753b-1774-45b0-b34b-6b06eb31a678\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") "
Sep 11 18:41:45.584295 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.584141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd8np\" (UniqueName: \"kubernetes.io/projected/3652753b-1774-45b0-b34b-6b06eb31a678-kube-api-access-rd8np\") pod \"3652753b-1774-45b0-b34b-6b06eb31a678\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") "
Sep 11 18:41:45.584295 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.584176    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-maas-api-tls\") pod \"3652753b-1774-45b0-b34b-6b06eb31a678\" (UID: \"3652753b-1774-45b0-b34b-6b06eb31a678\") "
Sep 11 18:41:45.584550 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.584514    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3652753b-1774-45b0-b34b-6b06eb31a678-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3652753b-1774-45b0-b34b-6b06eb31a678" (UID: "3652753b-1774-45b0-b34b-6b06eb31a678"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:45.586476 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.586447    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3652753b-1774-45b0-b34b-6b06eb31a678" (UID: "3652753b-1774-45b0-b34b-6b06eb31a678"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:45.586600 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.586491    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3652753b-1774-45b0-b34b-6b06eb31a678" (UID: "3652753b-1774-45b0-b34b-6b06eb31a678"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:45.586600 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.586499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3652753b-1774-45b0-b34b-6b06eb31a678-kube-api-access-rd8np" (OuterVolumeSpecName: "kube-api-access-rd8np") pod "3652753b-1774-45b0-b34b-6b06eb31a678" (UID: "3652753b-1774-45b0-b34b-6b06eb31a678"). InnerVolumeSpecName "kube-api-access-rd8np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:45.686080 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.686036    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3652753b-1774-45b0-b34b-6b06eb31a678-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:45.686080 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.686073    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:45.686584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.686088    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd8np\" (UniqueName: \"kubernetes.io/projected/3652753b-1774-45b0-b34b-6b06eb31a678-kube-api-access-rd8np\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:45.686584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.686103    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3652753b-1774-45b0-b34b-6b06eb31a678-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:45.730116 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.730054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:45.840077 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.839982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:45.891626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.891589    2580 generic.go:356] "Generic (PLEG): container finished" podID="3652753b-1774-45b0-b34b-6b06eb31a678" containerID="00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6" exitCode=0
Sep 11 18:41:45.891805 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.891635    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"
Sep 11 18:41:45.891805 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.891679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97" event={"ID":"3652753b-1774-45b0-b34b-6b06eb31a678","Type":"ContainerDied","Data":"00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6"}
Sep 11 18:41:45.891805 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.891739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97" event={"ID":"3652753b-1774-45b0-b34b-6b06eb31a678","Type":"ContainerDied","Data":"37b929b3c0155b717b3744f7d9c5cb317eb60ddcb1d052803d36e506d77c5a78"}
Sep 11 18:41:45.891805 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.891769    2580 scope.go:117] "RemoveContainer" containerID="00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6"
Sep 11 18:41:45.901909 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.901881    2580 scope.go:117] "RemoveContainer" containerID="00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6"
Sep 11 18:41:45.902284 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:41:45.902262    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6\": container with ID starting with 00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6 not found: ID does not exist" containerID="00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6"
Sep 11 18:41:45.902358 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.902297    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6"} err="failed to get container status \"00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6\": rpc error: code = NotFound desc = could not find container \"00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6\": container with ID starting with 00d155141fa8148a409d1fbfbbf05d97849441834458292c833309648370abf6 not found: ID does not exist"
Sep 11 18:41:45.920537 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.920491    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"]
Sep 11 18:41:45.931525 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:45.931488    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-efa8b9-84679cf8f-kns97"]
Sep 11 18:41:46.729904 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:46.729870    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:47.395868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:47.395832    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3652753b-1774-45b0-b34b-6b06eb31a678" path="/var/lib/kubelet/pods/3652753b-1774-45b0-b34b-6b06eb31a678/volumes"
Sep 11 18:41:47.543233 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:47.543176    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:48.310157 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:48.310119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:49.160695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:49.160656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:49.947564 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:49.947524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:51.854420 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:51.854379    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg" podUID="4eb8a648-6c34-40c6-b800-1d9d9e553aea" containerName="main" containerID="cri-o://cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764" gracePeriod=45
Sep 11 18:41:51.892079 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:51.892042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:51.997089 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:51.997057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:52.109051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.108973    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:41:52.164185 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.164143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:52.249357 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.249321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tls-certs\") pod \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") "
Sep 11 18:41:52.249564 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.249429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-model-cache\") pod \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") "
Sep 11 18:41:52.249564 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.249458    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6gkj\" (UniqueName: \"kubernetes.io/projected/4eb8a648-6c34-40c6-b800-1d9d9e553aea-kube-api-access-p6gkj\") pod \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") "
Sep 11 18:41:52.249564 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.249489    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-home\") pod \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") "
Sep 11 18:41:52.249564 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.249513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-dshm\") pod \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") "
Sep 11 18:41:52.249784 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.249714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tmp-dir\") pod \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\" (UID: \"4eb8a648-6c34-40c6-b800-1d9d9e553aea\") "
Sep 11 18:41:52.249848 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.249771    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-home" (OuterVolumeSpecName: "home") pod "4eb8a648-6c34-40c6-b800-1d9d9e553aea" (UID: "4eb8a648-6c34-40c6-b800-1d9d9e553aea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.249848 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.249784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-model-cache" (OuterVolumeSpecName: "model-cache") pod "4eb8a648-6c34-40c6-b800-1d9d9e553aea" (UID: "4eb8a648-6c34-40c6-b800-1d9d9e553aea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.250000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.249972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4eb8a648-6c34-40c6-b800-1d9d9e553aea" (UID: "4eb8a648-6c34-40c6-b800-1d9d9e553aea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.250293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.250272    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-model-cache\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.250438 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.250425    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-home\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.250559 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.250546    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tmp-dir\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.252500 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.252470    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb8a648-6c34-40c6-b800-1d9d9e553aea-kube-api-access-p6gkj" (OuterVolumeSpecName: "kube-api-access-p6gkj") pod "4eb8a648-6c34-40c6-b800-1d9d9e553aea" (UID: "4eb8a648-6c34-40c6-b800-1d9d9e553aea"). InnerVolumeSpecName "kube-api-access-p6gkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:52.252611 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.252494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-dshm" (OuterVolumeSpecName: "dshm") pod "4eb8a648-6c34-40c6-b800-1d9d9e553aea" (UID: "4eb8a648-6c34-40c6-b800-1d9d9e553aea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.252611 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.252552    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4eb8a648-6c34-40c6-b800-1d9d9e553aea" (UID: "4eb8a648-6c34-40c6-b800-1d9d9e553aea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:52.351443 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.351383    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb8a648-6c34-40c6-b800-1d9d9e553aea-tls-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.351443 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.351436    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6gkj\" (UniqueName: \"kubernetes.io/projected/4eb8a648-6c34-40c6-b800-1d9d9e553aea-kube-api-access-p6gkj\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.351443 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.351448    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb8a648-6c34-40c6-b800-1d9d9e553aea-dshm\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.937661 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.937626    2580 generic.go:356] "Generic (PLEG): container finished" podID="4eb8a648-6c34-40c6-b800-1d9d9e553aea" containerID="cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764" exitCode=0
Sep 11 18:41:52.937661 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.937663    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"
Sep 11 18:41:52.938223 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.937705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg" event={"ID":"4eb8a648-6c34-40c6-b800-1d9d9e553aea","Type":"ContainerDied","Data":"cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764"}
Sep 11 18:41:52.938223 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.937757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg" event={"ID":"4eb8a648-6c34-40c6-b800-1d9d9e553aea","Type":"ContainerDied","Data":"ea23a11944a7770e7cc2dda009247220c62f7131d9e56555b858a5a8b742ed3e"}
Sep 11 18:41:52.938223 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.937776    2580 scope.go:117] "RemoveContainer" containerID="cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764"
Sep 11 18:41:52.947256 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.947236    2580 scope.go:117] "RemoveContainer" containerID="cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764"
Sep 11 18:41:52.947611 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:41:52.947586    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764\": container with ID starting with cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764 not found: ID does not exist" containerID="cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764"
Sep 11 18:41:52.947696 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.947619    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764"} err="failed to get container status \"cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764\": rpc error: code = NotFound desc = could not find container \"cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764\": container with ID starting with cd9736458f46a554cf3ec37bd6bf195327a601bb8ce1edd1bfa20711b60e8764 not found: ID does not exist"
Sep 11 18:41:52.960985 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.960948    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"]
Sep 11 18:41:52.963293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:52.963260    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-2-w0-efa8b9-kserve-b66bc9f5-kc9qg"]
Sep 11 18:41:53.398640 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:53.398592    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb8a648-6c34-40c6-b800-1d9d9e553aea" path="/var/lib/kubelet/pods/4eb8a648-6c34-40c6-b800-1d9d9e553aea/volumes"
Sep 11 18:41:53.589688 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:53.589633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:53.944698 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:53.944660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:54.758899 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:54.758858    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"]
Sep 11 18:41:54.759170 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:54.759119    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk" podUID="91916eba-fb73-46cd-ab4a-6554531cbbba" containerName="maas-api" containerID="cri-o://24faf8afe8268b0fce04cc8abb56b2dae938cdb769a1b3b1dbcf5035d73c9180" gracePeriod=30
Sep 11 18:41:54.790661 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:54.790621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:54.878893 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:54.878851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:54.942425 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:54.942386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:54.953277 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:54.953240    2580 generic.go:356] "Generic (PLEG): container finished" podID="91916eba-fb73-46cd-ab4a-6554531cbbba" containerID="24faf8afe8268b0fce04cc8abb56b2dae938cdb769a1b3b1dbcf5035d73c9180" exitCode=0
Sep 11 18:41:54.953656 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:54.953367    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk" event={"ID":"91916eba-fb73-46cd-ab4a-6554531cbbba","Type":"ContainerDied","Data":"24faf8afe8268b0fce04cc8abb56b2dae938cdb769a1b3b1dbcf5035d73c9180"}
Sep 11 18:41:55.012404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.012334    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:41:55.045577 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.045475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:55.117652 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.117600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:55.180788 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.180739    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-maas-api-tls\") pod \"91916eba-fb73-46cd-ab4a-6554531cbbba\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") "
Sep 11 18:41:55.180788 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.180788    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r85nk\" (UniqueName: \"kubernetes.io/projected/91916eba-fb73-46cd-ab4a-6554531cbbba-kube-api-access-r85nk\") pod \"91916eba-fb73-46cd-ab4a-6554531cbbba\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") "
Sep 11 18:41:55.181050 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.180882    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91916eba-fb73-46cd-ab4a-6554531cbbba-openshift-service-ca\") pod \"91916eba-fb73-46cd-ab4a-6554531cbbba\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") "
Sep 11 18:41:55.181050 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.180908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-metrics-certs\") pod \"91916eba-fb73-46cd-ab4a-6554531cbbba\" (UID: \"91916eba-fb73-46cd-ab4a-6554531cbbba\") "
Sep 11 18:41:55.181283 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.181257    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91916eba-fb73-46cd-ab4a-6554531cbbba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91916eba-fb73-46cd-ab4a-6554531cbbba" (UID: "91916eba-fb73-46cd-ab4a-6554531cbbba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:55.183051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.183021    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "91916eba-fb73-46cd-ab4a-6554531cbbba" (UID: "91916eba-fb73-46cd-ab4a-6554531cbbba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:55.183051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.183040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91916eba-fb73-46cd-ab4a-6554531cbbba" (UID: "91916eba-fb73-46cd-ab4a-6554531cbbba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:55.183216 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.183119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91916eba-fb73-46cd-ab4a-6554531cbbba-kube-api-access-r85nk" (OuterVolumeSpecName: "kube-api-access-r85nk") pod "91916eba-fb73-46cd-ab4a-6554531cbbba" (UID: "91916eba-fb73-46cd-ab4a-6554531cbbba"). InnerVolumeSpecName "kube-api-access-r85nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:55.281855 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.281775    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.281855 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.281806    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r85nk\" (UniqueName: \"kubernetes.io/projected/91916eba-fb73-46cd-ab4a-6554531cbbba-kube-api-access-r85nk\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.281855 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.281816    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91916eba-fb73-46cd-ab4a-6554531cbbba-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.281855 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.281825    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91916eba-fb73-46cd-ab4a-6554531cbbba-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.425078 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.425039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:55.533897 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.533789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:55.963085 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.963048    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"
Sep 11 18:41:55.963085 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.963061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk" event={"ID":"91916eba-fb73-46cd-ab4a-6554531cbbba","Type":"ContainerDied","Data":"c27c36a40942408a8d23d59e5c07d630a7dc04dd12cd6f669188381bd9106446"}
Sep 11 18:41:55.963695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.963138    2580 scope.go:117] "RemoveContainer" containerID="24faf8afe8268b0fce04cc8abb56b2dae938cdb769a1b3b1dbcf5035d73c9180"
Sep 11 18:41:55.985790 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.985751    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"]
Sep 11 18:41:55.995923 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:55.995881    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ead89-7f9b4dd79c-ns7vk"]
Sep 11 18:41:56.466289 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:56.466251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:57.396459 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:57.396410    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91916eba-fb73-46cd-ab4a-6554531cbbba" path="/var/lib/kubelet/pods/91916eba-fb73-46cd-ab4a-6554531cbbba/volumes"
Sep 11 18:41:57.486824 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:57.486758    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:41:58.439814 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:41:58.439774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:01.741567 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:01.741522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:01.840499 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:01.840461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:02.025021 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:02.024935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:09.045569 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:09.045528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:09.144895 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:09.144849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:09.466677 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:09.466619    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:15.573297 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:15.573184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:16.750102 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:16.750065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:17.212309 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:17.212264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:18.174737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:18.174696    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:19.039051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:19.039006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:19.927823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:19.927787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:20.834346 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:20.834307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:21.717594 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:21.717552    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"]
Sep 11 18:42:21.750611 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:21.750573    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"]
Sep 11 18:42:21.810074 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:21.810034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:25.533147 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:25.533107    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"]
Sep 11 18:42:25.543557 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:25.543508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:26.856129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:26.856086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:32.605575 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:32.605537    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"]
Sep 11 18:42:32.605997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:32.605757    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k" podUID="5e8f0b7e-49db-4650-8152-923be3fd1278" containerName="maas-api" containerID="cri-o://0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458" gracePeriod=30
Sep 11 18:42:32.632917 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:32.632875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:32.728655 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:32.728568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:32.788781 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:32.788700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:32.865963 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:32.865895    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:42:32.893625 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:32.893573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:32.993504 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:32.993462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:33.052405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.052353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqctr\" (UniqueName: \"kubernetes.io/projected/5e8f0b7e-49db-4650-8152-923be3fd1278-kube-api-access-kqctr\") pod \"5e8f0b7e-49db-4650-8152-923be3fd1278\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") "
Sep 11 18:42:33.052617 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.052428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-maas-api-tls\") pod \"5e8f0b7e-49db-4650-8152-923be3fd1278\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") "
Sep 11 18:42:33.052617 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.052498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e8f0b7e-49db-4650-8152-923be3fd1278-openshift-service-ca\") pod \"5e8f0b7e-49db-4650-8152-923be3fd1278\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") "
Sep 11 18:42:33.052617 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.052578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-metrics-certs\") pod \"5e8f0b7e-49db-4650-8152-923be3fd1278\" (UID: \"5e8f0b7e-49db-4650-8152-923be3fd1278\") "
Sep 11 18:42:33.053295 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.053235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8f0b7e-49db-4650-8152-923be3fd1278-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e8f0b7e-49db-4650-8152-923be3fd1278" (UID: "5e8f0b7e-49db-4650-8152-923be3fd1278"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:42:33.054858 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.054831    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e8f0b7e-49db-4650-8152-923be3fd1278" (UID: "5e8f0b7e-49db-4650-8152-923be3fd1278"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:42:33.054957 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.054838    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8f0b7e-49db-4650-8152-923be3fd1278-kube-api-access-kqctr" (OuterVolumeSpecName: "kube-api-access-kqctr") pod "5e8f0b7e-49db-4650-8152-923be3fd1278" (UID: "5e8f0b7e-49db-4650-8152-923be3fd1278"). InnerVolumeSpecName "kube-api-access-kqctr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:33.055131 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.055111    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5e8f0b7e-49db-4650-8152-923be3fd1278" (UID: "5e8f0b7e-49db-4650-8152-923be3fd1278"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:42:33.153885 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.153844    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:33.153885 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.153881    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqctr\" (UniqueName: \"kubernetes.io/projected/5e8f0b7e-49db-4650-8152-923be3fd1278-kube-api-access-kqctr\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:33.154109 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.153897    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e8f0b7e-49db-4650-8152-923be3fd1278-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:33.154109 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.153911    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e8f0b7e-49db-4650-8152-923be3fd1278-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:33.169466 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.169422    2580 generic.go:356] "Generic (PLEG): container finished" podID="5e8f0b7e-49db-4650-8152-923be3fd1278" containerID="0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458" exitCode=0
Sep 11 18:42:33.169466 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.169471    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"
Sep 11 18:42:33.169686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.169512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k" event={"ID":"5e8f0b7e-49db-4650-8152-923be3fd1278","Type":"ContainerDied","Data":"0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458"}
Sep 11 18:42:33.169686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.169569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k" event={"ID":"5e8f0b7e-49db-4650-8152-923be3fd1278","Type":"ContainerDied","Data":"62f55dd595ae1307e77587f275c8458b8a938c36519de517b8d9da33e77a8502"}
Sep 11 18:42:33.169686 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.169592    2580 scope.go:117] "RemoveContainer" containerID="0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458"
Sep 11 18:42:33.184559 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.184533    2580 scope.go:117] "RemoveContainer" containerID="0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458"
Sep 11 18:42:33.184974 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:42:33.184942    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458\": container with ID starting with 0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458 not found: ID does not exist" containerID="0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458"
Sep 11 18:42:33.185069 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.184987    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458"} err="failed to get container status \"0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458\": rpc error: code = NotFound desc = could not find container \"0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458\": container with ID starting with 0c5a6598e2ff4e677463fff646aea2672ebb0c2b12c16b19e8ac161364137458 not found: ID does not exist"
Sep 11 18:42:33.200091 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.200048    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"]
Sep 11 18:42:33.207673 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.207637    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-724327-86788c7b6f-c227k"]
Sep 11 18:42:33.245513 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.245470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:33.360676 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.360632    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:33.396230 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:33.396172    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e8f0b7e-49db-4650-8152-923be3fd1278" path="/var/lib/kubelet/pods/5e8f0b7e-49db-4650-8152-923be3fd1278/volumes"
Sep 11 18:42:34.172773 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:34.172733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:34.966862 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:34.966828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:36.737180 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:36.737140    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r" podUID="936eb4fd-d5ae-4c12-94df-90884cd7e856" containerName="main" containerID="cri-o://b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4" gracePeriod=45
Sep 11 18:42:36.770106 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:36.770067    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr" podUID="1851c24f-43b2-48be-bcba-8471fa8785af" containerName="main" containerID="cri-o://1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b" gracePeriod=45
Sep 11 18:42:37.063624 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.063596    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:42:37.067395 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.067363    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:42:37.093032 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093001    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-home\") pod \"936eb4fd-d5ae-4c12-94df-90884cd7e856\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") "
Sep 11 18:42:37.093240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093043    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1851c24f-43b2-48be-bcba-8471fa8785af-tls-certs\") pod \"1851c24f-43b2-48be-bcba-8471fa8785af\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") "
Sep 11 18:42:37.093240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093073    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-dshm\") pod \"1851c24f-43b2-48be-bcba-8471fa8785af\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") "
Sep 11 18:42:37.093240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093089    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xsf2\" (UniqueName: \"kubernetes.io/projected/936eb4fd-d5ae-4c12-94df-90884cd7e856-kube-api-access-7xsf2\") pod \"936eb4fd-d5ae-4c12-94df-90884cd7e856\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") "
Sep 11 18:42:37.093240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093128    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-tmp-dir\") pod \"936eb4fd-d5ae-4c12-94df-90884cd7e856\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") "
Sep 11 18:42:37.093240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093152    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-home\") pod \"1851c24f-43b2-48be-bcba-8471fa8785af\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") "
Sep 11 18:42:37.093539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093241    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-tmp-dir\") pod \"1851c24f-43b2-48be-bcba-8471fa8785af\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") "
Sep 11 18:42:37.093539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093285    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936eb4fd-d5ae-4c12-94df-90884cd7e856-tls-certs\") pod \"936eb4fd-d5ae-4c12-94df-90884cd7e856\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") "
Sep 11 18:42:37.093539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-dshm\") pod \"936eb4fd-d5ae-4c12-94df-90884cd7e856\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") "
Sep 11 18:42:37.093539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093364    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24dqr\" (UniqueName: \"kubernetes.io/projected/1851c24f-43b2-48be-bcba-8471fa8785af-kube-api-access-24dqr\") pod \"1851c24f-43b2-48be-bcba-8471fa8785af\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") "
Sep 11 18:42:37.093539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-model-cache\") pod \"1851c24f-43b2-48be-bcba-8471fa8785af\" (UID: \"1851c24f-43b2-48be-bcba-8471fa8785af\") "
Sep 11 18:42:37.093539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093422    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-model-cache\") pod \"936eb4fd-d5ae-4c12-94df-90884cd7e856\" (UID: \"936eb4fd-d5ae-4c12-94df-90884cd7e856\") "
Sep 11 18:42:37.093539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093276    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-home" (OuterVolumeSpecName: "home") pod "936eb4fd-d5ae-4c12-94df-90884cd7e856" (UID: "936eb4fd-d5ae-4c12-94df-90884cd7e856"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.093539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093480    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "936eb4fd-d5ae-4c12-94df-90884cd7e856" (UID: "936eb4fd-d5ae-4c12-94df-90884cd7e856"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.093954 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093721    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-tmp-dir\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.093954 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093739    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-home\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.093954 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-model-cache" (OuterVolumeSpecName: "model-cache") pod "1851c24f-43b2-48be-bcba-8471fa8785af" (UID: "1851c24f-43b2-48be-bcba-8471fa8785af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.093954 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.093861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-model-cache" (OuterVolumeSpecName: "model-cache") pod "936eb4fd-d5ae-4c12-94df-90884cd7e856" (UID: "936eb4fd-d5ae-4c12-94df-90884cd7e856"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.094251 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.094098    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1851c24f-43b2-48be-bcba-8471fa8785af" (UID: "1851c24f-43b2-48be-bcba-8471fa8785af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.094251 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.094244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-home" (OuterVolumeSpecName: "home") pod "1851c24f-43b2-48be-bcba-8471fa8785af" (UID: "1851c24f-43b2-48be-bcba-8471fa8785af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.095947 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.095899    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1851c24f-43b2-48be-bcba-8471fa8785af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1851c24f-43b2-48be-bcba-8471fa8785af" (UID: "1851c24f-43b2-48be-bcba-8471fa8785af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:42:37.096312 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.096226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-dshm" (OuterVolumeSpecName: "dshm") pod "1851c24f-43b2-48be-bcba-8471fa8785af" (UID: "1851c24f-43b2-48be-bcba-8471fa8785af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.096493 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.096441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936eb4fd-d5ae-4c12-94df-90884cd7e856-kube-api-access-7xsf2" (OuterVolumeSpecName: "kube-api-access-7xsf2") pod "936eb4fd-d5ae-4c12-94df-90884cd7e856" (UID: "936eb4fd-d5ae-4c12-94df-90884cd7e856"). InnerVolumeSpecName "kube-api-access-7xsf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:37.096493 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.096476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1851c24f-43b2-48be-bcba-8471fa8785af-kube-api-access-24dqr" (OuterVolumeSpecName: "kube-api-access-24dqr") pod "1851c24f-43b2-48be-bcba-8471fa8785af" (UID: "1851c24f-43b2-48be-bcba-8471fa8785af"). InnerVolumeSpecName "kube-api-access-24dqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:37.096877 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.096850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936eb4fd-d5ae-4c12-94df-90884cd7e856-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "936eb4fd-d5ae-4c12-94df-90884cd7e856" (UID: "936eb4fd-d5ae-4c12-94df-90884cd7e856"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:42:37.098644 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.098617    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-dshm" (OuterVolumeSpecName: "dshm") pod "936eb4fd-d5ae-4c12-94df-90884cd7e856" (UID: "936eb4fd-d5ae-4c12-94df-90884cd7e856"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.191760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.191723    2580 generic.go:356] "Generic (PLEG): container finished" podID="936eb4fd-d5ae-4c12-94df-90884cd7e856" containerID="b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4" exitCode=0
Sep 11 18:42:37.191963 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.191793    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"
Sep 11 18:42:37.191963 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.191819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r" event={"ID":"936eb4fd-d5ae-4c12-94df-90884cd7e856","Type":"ContainerDied","Data":"b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4"}
Sep 11 18:42:37.191963 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.191869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r" event={"ID":"936eb4fd-d5ae-4c12-94df-90884cd7e856","Type":"ContainerDied","Data":"34fcff8b93838d166125607333376a32d045004c6e4403e37c2958a2af483385"}
Sep 11 18:42:37.191963 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.191891    2580 scope.go:117] "RemoveContainer" containerID="b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4"
Sep 11 18:42:37.193404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.193381    2580 generic.go:356] "Generic (PLEG): container finished" podID="1851c24f-43b2-48be-bcba-8471fa8785af" containerID="1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b" exitCode=0
Sep 11 18:42:37.193516 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.193420    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"
Sep 11 18:42:37.193516 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.193421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr" event={"ID":"1851c24f-43b2-48be-bcba-8471fa8785af","Type":"ContainerDied","Data":"1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b"}
Sep 11 18:42:37.193516 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.193453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr" event={"ID":"1851c24f-43b2-48be-bcba-8471fa8785af","Type":"ContainerDied","Data":"fdeaa2b8ba186592a74e89d7cfb73c6da8473f0da098559e2efbf63e7daf2c3e"}
Sep 11 18:42:37.194286 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.194254    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1851c24f-43b2-48be-bcba-8471fa8785af-tls-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.194286 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.194281    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-dshm\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.194432 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.194294    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xsf2\" (UniqueName: \"kubernetes.io/projected/936eb4fd-d5ae-4c12-94df-90884cd7e856-kube-api-access-7xsf2\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.194432 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.194306    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-home\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.194432 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.194316    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-tmp-dir\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.194432 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.194327    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936eb4fd-d5ae-4c12-94df-90884cd7e856-tls-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.194432 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.194338    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-dshm\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.194432 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.194350    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dqr\" (UniqueName: \"kubernetes.io/projected/1851c24f-43b2-48be-bcba-8471fa8785af-kube-api-access-24dqr\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.194432 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.194362    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1851c24f-43b2-48be-bcba-8471fa8785af-model-cache\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.194432 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.194375    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936eb4fd-d5ae-4c12-94df-90884cd7e856-model-cache\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.201056 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.201030    2580 scope.go:117] "RemoveContainer" containerID="b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4"
Sep 11 18:42:37.201437 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:42:37.201414    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4\": container with ID starting with b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4 not found: ID does not exist" containerID="b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4"
Sep 11 18:42:37.201551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.201448    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4"} err="failed to get container status \"b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4\": rpc error: code = NotFound desc = could not find container \"b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4\": container with ID starting with b9a6c0fcffe1622ae770908a0a66f8b6fe2bafa6cb2465fa5e36b076cf7145a4 not found: ID does not exist"
Sep 11 18:42:37.201551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.201472    2580 scope.go:117] "RemoveContainer" containerID="1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b"
Sep 11 18:42:37.211544 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.211522    2580 scope.go:117] "RemoveContainer" containerID="1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b"
Sep 11 18:42:37.211914 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:42:37.211890    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b\": container with ID starting with 1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b not found: ID does not exist" containerID="1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b"
Sep 11 18:42:37.212000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.211920    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b"} err="failed to get container status \"1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b\": rpc error: code = NotFound desc = could not find container \"1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b\": container with ID starting with 1fb39b2d2ff1c6743231f4fa868e34049bdc1ebd6713897f91f761380a75eb3b not found: ID does not exist"
Sep 11 18:42:37.225872 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.225832    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"]
Sep 11 18:42:37.230183 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.230151    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-724327/simulator-w4-724327-kserve-66b98d66f6-kj2nr"]
Sep 11 18:42:37.242935 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.242897    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"]
Sep 11 18:42:37.251840 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.251808    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-2-w4-724327-kserve-5bd8bf858c-bqm7r"]
Sep 11 18:42:37.401408 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.401371    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1851c24f-43b2-48be-bcba-8471fa8785af" path="/var/lib/kubelet/pods/1851c24f-43b2-48be-bcba-8471fa8785af/volumes"
Sep 11 18:42:37.401772 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:37.401759    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="936eb4fd-d5ae-4c12-94df-90884cd7e856" path="/var/lib/kubelet/pods/936eb4fd-d5ae-4c12-94df-90884cd7e856/volumes"
Sep 11 18:42:40.289303 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.289257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:40.387665 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.387628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:40.551795 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.551707    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"]
Sep 11 18:42:40.551955 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.551919    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy" containerID="cri-o://791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56" gracePeriod=30
Sep 11 18:42:40.554595 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.554559    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v" podUID="a55d04b7-206c-48e9-be24-41da4bbc808d" containerName="main" containerID="cri-o://7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d" gracePeriod=45
Sep 11 18:42:40.564384 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.564345    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:40.805153 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.805071    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:42:40.831189 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.831150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-model-cache\") pod \"a55d04b7-206c-48e9-be24-41da4bbc808d\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") "
Sep 11 18:42:40.831404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.831228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx8xk\" (UniqueName: \"kubernetes.io/projected/a55d04b7-206c-48e9-be24-41da4bbc808d-kube-api-access-gx8xk\") pod \"a55d04b7-206c-48e9-be24-41da4bbc808d\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") "
Sep 11 18:42:40.831404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.831280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-dshm\") pod \"a55d04b7-206c-48e9-be24-41da4bbc808d\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") "
Sep 11 18:42:40.831404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.831361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55d04b7-206c-48e9-be24-41da4bbc808d-tls-certs\") pod \"a55d04b7-206c-48e9-be24-41da4bbc808d\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") "
Sep 11 18:42:40.831404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.831392    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-tmp-dir\") pod \"a55d04b7-206c-48e9-be24-41da4bbc808d\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") "
Sep 11 18:42:40.831619 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.831427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-home\") pod \"a55d04b7-206c-48e9-be24-41da4bbc808d\" (UID: \"a55d04b7-206c-48e9-be24-41da4bbc808d\") "
Sep 11 18:42:40.831619 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.831495    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a55d04b7-206c-48e9-be24-41da4bbc808d" (UID: "a55d04b7-206c-48e9-be24-41da4bbc808d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:40.831717 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.831664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a55d04b7-206c-48e9-be24-41da4bbc808d" (UID: "a55d04b7-206c-48e9-be24-41da4bbc808d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:40.831774 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.831726    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-model-cache\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:40.831826 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.831795    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-home" (OuterVolumeSpecName: "home") pod "a55d04b7-206c-48e9-be24-41da4bbc808d" (UID: "a55d04b7-206c-48e9-be24-41da4bbc808d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:40.833579 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.833553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-dshm" (OuterVolumeSpecName: "dshm") pod "a55d04b7-206c-48e9-be24-41da4bbc808d" (UID: "a55d04b7-206c-48e9-be24-41da4bbc808d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:40.833737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.833672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55d04b7-206c-48e9-be24-41da4bbc808d-kube-api-access-gx8xk" (OuterVolumeSpecName: "kube-api-access-gx8xk") pod "a55d04b7-206c-48e9-be24-41da4bbc808d" (UID: "a55d04b7-206c-48e9-be24-41da4bbc808d"). InnerVolumeSpecName "kube-api-access-gx8xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:40.834066 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.834048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55d04b7-206c-48e9-be24-41da4bbc808d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a55d04b7-206c-48e9-be24-41da4bbc808d" (UID: "a55d04b7-206c-48e9-be24-41da4bbc808d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:42:40.932631 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.932589    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-tmp-dir\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:40.932631 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.932621    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-home\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:40.932631 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.932633    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx8xk\" (UniqueName: \"kubernetes.io/projected/a55d04b7-206c-48e9-be24-41da4bbc808d-kube-api-access-gx8xk\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:40.932885 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.932645    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55d04b7-206c-48e9-be24-41da4bbc808d-dshm\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:40.932885 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:40.932658    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55d04b7-206c-48e9-be24-41da4bbc808d-tls-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:41.219877 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:41.219841    2580 generic.go:356] "Generic (PLEG): container finished" podID="a55d04b7-206c-48e9-be24-41da4bbc808d" containerID="7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d" exitCode=0
Sep 11 18:42:41.220101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:41.219883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"
Sep 11 18:42:41.220101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:41.219888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v" event={"ID":"a55d04b7-206c-48e9-be24-41da4bbc808d","Type":"ContainerDied","Data":"7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d"}
Sep 11 18:42:41.220101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:41.219991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v" event={"ID":"a55d04b7-206c-48e9-be24-41da4bbc808d","Type":"ContainerDied","Data":"bb7dc7acc9b49c6d626cea8bf34cb381ea79d3c32334f4947c060b14c7295a23"}
Sep 11 18:42:41.220101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:41.220016    2580 scope.go:117] "RemoveContainer" containerID="7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d"
Sep 11 18:42:41.229416 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:41.229388    2580 scope.go:117] "RemoveContainer" containerID="7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d"
Sep 11 18:42:41.229780 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:42:41.229759    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d\": container with ID starting with 7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d not found: ID does not exist" containerID="7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d"
Sep 11 18:42:41.229832 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:41.229790    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d"} err="failed to get container status \"7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d\": rpc error: code = NotFound desc = could not find container \"7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d\": container with ID starting with 7f792a8b3b8cf3e4e07edcc97676b28bcc1518dc87861aa1dbf431ba4c574e4d not found: ID does not exist"
Sep 11 18:42:41.246739 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:41.246692    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"]
Sep 11 18:42:41.249588 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:41.249559    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/auth-test-model-7ca6df-kserve-54d48874ff-5697v"]
Sep 11 18:42:41.396506 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:41.396469    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a55d04b7-206c-48e9-be24-41da4bbc808d" path="/var/lib/kubelet/pods/a55d04b7-206c-48e9-be24-41da4bbc808d/volumes"
Sep 11 18:42:45.801770 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.801739    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:42:45.873037 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873000    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istiod-ca-cert\") pod \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") "
Sep 11 18:42:45.873221 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873060    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-envoy\") pod \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") "
Sep 11 18:42:45.873221 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-data\") pod \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") "
Sep 11 18:42:45.873221 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873134    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-podinfo\") pod \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") "
Sep 11 18:42:45.873221 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873172    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-certs\") pod \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") "
Sep 11 18:42:45.873446 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873230    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-credential-socket\") pod \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") "
Sep 11 18:42:45.873446 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-token\") pod \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") "
Sep 11 18:42:45.873446 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-socket\") pod \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") "
Sep 11 18:42:45.873446 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4jgp\" (UniqueName: \"kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-kube-api-access-q4jgp\") pod \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\" (UID: \"d49cbaa6-1faa-45ed-8577-f474aeb73a9b\") "
Sep 11 18:42:45.873655 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-data" (OuterVolumeSpecName: "istio-data") pod "d49cbaa6-1faa-45ed-8577-f474aeb73a9b" (UID: "d49cbaa6-1faa-45ed-8577-f474aeb73a9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:45.873655 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d49cbaa6-1faa-45ed-8577-f474aeb73a9b" (UID: "d49cbaa6-1faa-45ed-8577-f474aeb73a9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:42:45.873655 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873581    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d49cbaa6-1faa-45ed-8577-f474aeb73a9b" (UID: "d49cbaa6-1faa-45ed-8577-f474aeb73a9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:45.873655 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d49cbaa6-1faa-45ed-8577-f474aeb73a9b" (UID: "d49cbaa6-1faa-45ed-8577-f474aeb73a9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:45.873866 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873689    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istiod-ca-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:45.873866 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873707    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-data\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:45.873866 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873720    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:45.873866 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873732    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-credential-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:45.873866 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.873751    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d49cbaa6-1faa-45ed-8577-f474aeb73a9b" (UID: "d49cbaa6-1faa-45ed-8577-f474aeb73a9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:45.875412 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.875386    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d49cbaa6-1faa-45ed-8577-f474aeb73a9b" (UID: "d49cbaa6-1faa-45ed-8577-f474aeb73a9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:45.875609 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.875593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-kube-api-access-q4jgp" (OuterVolumeSpecName: "kube-api-access-q4jgp") pod "d49cbaa6-1faa-45ed-8577-f474aeb73a9b" (UID: "d49cbaa6-1faa-45ed-8577-f474aeb73a9b"). InnerVolumeSpecName "kube-api-access-q4jgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:45.875928 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.875906    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d49cbaa6-1faa-45ed-8577-f474aeb73a9b" (UID: "d49cbaa6-1faa-45ed-8577-f474aeb73a9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:45.875928 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.875913    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d49cbaa6-1faa-45ed-8577-f474aeb73a9b" (UID: "d49cbaa6-1faa-45ed-8577-f474aeb73a9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:42:45.974844 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.974799    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-podinfo\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:45.974844 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.974835    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-token\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:45.974844 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.974844    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-workload-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:45.974844 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.974852    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4jgp\" (UniqueName: \"kubernetes.io/projected/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-kube-api-access-q4jgp\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:45.975125 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:45.974862    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49cbaa6-1faa-45ed-8577-f474aeb73a9b-istio-envoy\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:42:46.246444 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.246343    2580 generic.go:356] "Generic (PLEG): container finished" podID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerID="791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56" exitCode=0
Sep 11 18:42:46.246444 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.246399    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"
Sep 11 18:42:46.246444 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.246430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" event={"ID":"d49cbaa6-1faa-45ed-8577-f474aeb73a9b","Type":"ContainerDied","Data":"791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56"}
Sep 11 18:42:46.246705 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.246488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7" event={"ID":"d49cbaa6-1faa-45ed-8577-f474aeb73a9b","Type":"ContainerDied","Data":"c4917a38bfcccf44e47a66a3e78778106bb159c06a181482beccc7fd30752513"}
Sep 11 18:42:46.246705 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.246511    2580 scope.go:117] "RemoveContainer" containerID="791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56"
Sep 11 18:42:46.255611 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.255583    2580 scope.go:117] "RemoveContainer" containerID="23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e"
Sep 11 18:42:46.265857 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.265833    2580 scope.go:117] "RemoveContainer" containerID="791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56"
Sep 11 18:42:46.266208 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:42:46.266175    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56\": container with ID starting with 791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56 not found: ID does not exist" containerID="791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56"
Sep 11 18:42:46.266275 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.266223    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56"} err="failed to get container status \"791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56\": rpc error: code = NotFound desc = could not find container \"791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56\": container with ID starting with 791b47804e4629e5cd8288c6c20cafbe53249bab9ae96565de5ed368df9b8a56 not found: ID does not exist"
Sep 11 18:42:46.266275 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.266245    2580 scope.go:117] "RemoveContainer" containerID="23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e"
Sep 11 18:42:46.266508 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:42:46.266493    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e\": container with ID starting with 23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e not found: ID does not exist" containerID="23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e"
Sep 11 18:42:46.266550 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.266514    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e"} err="failed to get container status \"23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e\": rpc error: code = NotFound desc = could not find container \"23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e\": container with ID starting with 23fcc3b1f1db10428c263ed738cf88176d19b23215b1288cee14a170e2783b6e not found: ID does not exist"
Sep 11 18:42:46.275880 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.275845    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"]
Sep 11 18:42:46.290658 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:46.290621    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-724327-openshift-default-794d8997c4-8l8r7"]
Sep 11 18:42:47.396275 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:47.396243    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" path="/var/lib/kubelet/pods/d49cbaa6-1faa-45ed-8577-f474aeb73a9b/volumes"
Sep 11 18:42:57.389820 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:57.389775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:58.019101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:58.019059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:42:58.117971 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:42:58.117934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:43:01.124501 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:43:01.124467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:43:06.756296 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:43:06.756259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:43:17.490136 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:43:17.490073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:43:24.951758 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:43:24.951722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:43:35.157681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:43:35.157636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:43:38.808574 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:43:38.808528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:43:38.908810 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:43:38.908775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:44:22.353885 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:22.353846    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:44:23.305578 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:23.305535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:44:32.043846 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:32.043801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:44:32.098353 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:32.098317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:44:35.444475 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.444432    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"]
Sep 11 18:44:35.444965 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.444948    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3652753b-1774-45b0-b34b-6b06eb31a678" containerName="maas-api"
Sep 11 18:44:35.445011 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.444970    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3652753b-1774-45b0-b34b-6b06eb31a678" containerName="maas-api"
Sep 11 18:44:35.445011 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.444984    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875cef81-64fb-4faa-8727-bde9c50e80f5" containerName="maas-api"
Sep 11 18:44:35.445011 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.444994    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="875cef81-64fb-4faa-8727-bde9c50e80f5" containerName="maas-api"
Sep 11 18:44:35.445011 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445009    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91916eba-fb73-46cd-ab4a-6554531cbbba" containerName="maas-api"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445019    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91916eba-fb73-46cd-ab4a-6554531cbbba" containerName="maas-api"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445033    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerName="istio-proxy"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445043    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerName="istio-proxy"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445063    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1851c24f-43b2-48be-bcba-8471fa8785af" containerName="main"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445071    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1851c24f-43b2-48be-bcba-8471fa8785af" containerName="main"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445087    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9edaf227-3348-429c-a1a0-e99873010fcd" containerName="maas-api"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445096    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9edaf227-3348-429c-a1a0-e99873010fcd" containerName="maas-api"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445108    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a55d04b7-206c-48e9-be24-41da4bbc808d" containerName="main"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445117    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a55d04b7-206c-48e9-be24-41da4bbc808d" containerName="main"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445130    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8f0b7e-49db-4650-8152-923be3fd1278" containerName="maas-api"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445138    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8f0b7e-49db-4650-8152-923be3fd1278" containerName="maas-api"
Sep 11 18:44:35.445148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445147    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936eb4fd-d5ae-4c12-94df-90884cd7e856" containerName="main"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445156    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="936eb4fd-d5ae-4c12-94df-90884cd7e856" containerName="main"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445165    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb8a648-6c34-40c6-b800-1d9d9e553aea" containerName="main"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445173    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb8a648-6c34-40c6-b800-1d9d9e553aea" containerName="main"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445185    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445194    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445224    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445232    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445320    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerName="istio-proxy"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445333    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3652753b-1774-45b0-b34b-6b06eb31a678" containerName="maas-api"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445341    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb8a648-6c34-40c6-b800-1d9d9e553aea" containerName="main"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445350    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9edaf227-3348-429c-a1a0-e99873010fcd" containerName="maas-api"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445360    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="936eb4fd-d5ae-4c12-94df-90884cd7e856" containerName="main"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445369    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerName="istio-proxy"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445377    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91916eba-fb73-46cd-ab4a-6554531cbbba" containerName="maas-api"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445386    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1851c24f-43b2-48be-bcba-8471fa8785af" containerName="main"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445394    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a55d04b7-206c-48e9-be24-41da4bbc808d" containerName="main"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445403    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445411    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8f0b7e-49db-4650-8152-923be3fd1278" containerName="maas-api"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445419    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="875cef81-64fb-4faa-8727-bde9c50e80f5" containerName="maas-api"
Sep 11 18:44:35.445527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.445429    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49cbaa6-1faa-45ed-8577-f474aeb73a9b" containerName="istio-proxy"
Sep 11 18:44:35.448615 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.448591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.451693 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.451664    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"openshift-service-ca.crt\""
Sep 11 18:44:35.451693 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.451682    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"rate-test-model-7ca6df-kserve-self-signed-certs\""
Sep 11 18:44:35.451916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.451687    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"default-dockercfg-9lxk4\""
Sep 11 18:44:35.451916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.451720    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"kube-root-ca.crt\""
Sep 11 18:44:35.456942 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.456914    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"]
Sep 11 18:44:35.476910 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.476869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:44:35.565413 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.565373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b61fa4-338f-418a-83c7-9e3a6053523f-tls-certs\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.565598 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.565439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-home\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.565598 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.565467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-model-cache\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.565598 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.565486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-tmp-dir\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.565598 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.565558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-dshm\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.565744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.565611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7kfp\" (UniqueName: \"kubernetes.io/projected/39b61fa4-338f-418a-83c7-9e3a6053523f-kube-api-access-d7kfp\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.666824 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.666780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-home\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.666824 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.666831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-model-cache\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.667088 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.666850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-tmp-dir\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.667088 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.666868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-dshm\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.667088 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.666918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kfp\" (UniqueName: \"kubernetes.io/projected/39b61fa4-338f-418a-83c7-9e3a6053523f-kube-api-access-d7kfp\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.667088 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.666982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b61fa4-338f-418a-83c7-9e3a6053523f-tls-certs\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.667320 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.667258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-tmp-dir\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.667372 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.667346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-model-cache\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.667424 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.667368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-home\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.669302 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.669278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-dshm\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.669620 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.669598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b61fa4-338f-418a-83c7-9e3a6053523f-tls-certs\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.679066 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.679039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7kfp\" (UniqueName: \"kubernetes.io/projected/39b61fa4-338f-418a-83c7-9e3a6053523f-kube-api-access-d7kfp\") pod \"rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:35.759160 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:35.759061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:36.029938 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:36.029894    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"]
Sep 11 18:44:36.032738 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:44:36.032709    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39b61fa4_338f_418a_83c7_9e3a6053523f.slice/crio-35d580522ecac4b48f74113c18c74d8d7304e5a0d3d450a4f4e26e73c88f873a WatchSource:0}: Error finding container 35d580522ecac4b48f74113c18c74d8d7304e5a0d3d450a4f4e26e73c88f873a: Status 404 returned error can't find the container with id 35d580522ecac4b48f74113c18c74d8d7304e5a0d3d450a4f4e26e73c88f873a
Sep 11 18:44:36.756932 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:36.756893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk" event={"ID":"39b61fa4-338f-418a-83c7-9e3a6053523f","Type":"ContainerStarted","Data":"9671f95d77818d85af952afc9e8d7d835cc0083e48cc00c21af817dea7bfecac"}
Sep 11 18:44:36.756932 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:36.756941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk" event={"ID":"39b61fa4-338f-418a-83c7-9e3a6053523f","Type":"ContainerStarted","Data":"35d580522ecac4b48f74113c18c74d8d7304e5a0d3d450a4f4e26e73c88f873a"}
Sep 11 18:44:36.774800 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:36.774730    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk" podStartSLOduration=1.774711454 podStartE2EDuration="1.774711454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:36.773075618 +0000 UTC m=+1757.869669249" watchObservedRunningTime="2026-09-11 18:44:36.774711454 +0000 UTC m=+1757.871305083"
Sep 11 18:44:45.760101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:45.760058    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:45.760530 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:45.760121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:45.772664 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:45.772628    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:45.806113 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:45.806085    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:44:53.177289 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:44:53.177225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:00.138006 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:00.137969    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q"]
Sep 11 18:45:00.138495 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:00.138313    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerName="istio-proxy"
Sep 11 18:45:00.138495 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:00.138324    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33acb3bf-828a-4380-9643-b0d3db926b3b" containerName="istio-proxy"
Sep 11 18:45:00.141402 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:00.141383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q"
Sep 11 18:45:00.143840 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:00.143817    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:45:00.149356 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:00.149319    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q"]
Sep 11 18:45:00.154063 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:00.154035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q"
Sep 11 18:45:00.423905 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:00.423880    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q"]
Sep 11 18:45:00.426726 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:45:00.426694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e14f13_ba29_4ad0_86c6_fda8abbf4b43.slice/crio-727e6d6a3fd15cb5dc7d7e7d65032ae7634a1c51a431f387e10b24f931ab6782 WatchSource:0}: Error finding container 727e6d6a3fd15cb5dc7d7e7d65032ae7634a1c51a431f387e10b24f931ab6782: Status 404 returned error can't find the container with id 727e6d6a3fd15cb5dc7d7e7d65032ae7634a1c51a431f387e10b24f931ab6782
Sep 11 18:45:00.429255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:00.429236    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:45:00.861574 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:00.861482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q" event={"ID":"32e14f13-ba29-4ad0-86c6-fda8abbf4b43","Type":"ContainerStarted","Data":"727e6d6a3fd15cb5dc7d7e7d65032ae7634a1c51a431f387e10b24f931ab6782"}
Sep 11 18:45:03.880008 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:03.879970    2580 generic.go:356] "Generic (PLEG): container finished" podID="32e14f13-ba29-4ad0-86c6-fda8abbf4b43" containerID="539656bd71b26de6a3737e96ad8c5b46168b69e147e1f3cdc0fdc13654924d8f" exitCode=0
Sep 11 18:45:03.880427 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:03.880059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q" event={"ID":"32e14f13-ba29-4ad0-86c6-fda8abbf4b43","Type":"ContainerDied","Data":"539656bd71b26de6a3737e96ad8c5b46168b69e147e1f3cdc0fdc13654924d8f"}
Sep 11 18:45:05.017776 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:05.017737    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q"
Sep 11 18:45:05.892217 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:05.892177    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q"
Sep 11 18:45:05.892401 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:05.892171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q" event={"ID":"32e14f13-ba29-4ad0-86c6-fda8abbf4b43","Type":"ContainerDied","Data":"727e6d6a3fd15cb5dc7d7e7d65032ae7634a1c51a431f387e10b24f931ab6782"}
Sep 11 18:45:05.892401 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:05.892340    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="727e6d6a3fd15cb5dc7d7e7d65032ae7634a1c51a431f387e10b24f931ab6782"
Sep 11 18:45:36.935193 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:36.935142    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"]
Sep 11 18:45:36.954035 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:36.953992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:37.869725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:37.869677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:38.281491 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.281447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:38.552166 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.552080    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"]
Sep 11 18:45:38.552450 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.552436    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e14f13-ba29-4ad0-86c6-fda8abbf4b43" containerName="cleanup"
Sep 11 18:45:38.552501 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.552453    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e14f13-ba29-4ad0-86c6-fda8abbf4b43" containerName="cleanup"
Sep 11 18:45:38.552536 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.552518    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e14f13-ba29-4ad0-86c6-fda8abbf4b43" containerName="cleanup"
Sep 11 18:45:38.559795 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.559768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.562092 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.562073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b0293c-openshift-default-dockercfg-vkcnf\""
Sep 11 18:45:38.568624 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.568587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"]
Sep 11 18:45:38.575736 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.575681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:38.650219 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.650155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-envoy\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.650219 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.650220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-socket\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.650497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.650268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-data\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.650497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.650305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-credential-socket\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.650497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.650331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istiod-ca-cert\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.650497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.650357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lzd\" (UniqueName: \"kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-kube-api-access-p2lzd\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.650497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.650389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-certs\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.650497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.650459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-podinfo\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.650497 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.650492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-token\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.751288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-envoy\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.751288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-socket\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.751535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-data\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.751535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-credential-socket\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.751535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istiod-ca-cert\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.751535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lzd\" (UniqueName: \"kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-kube-api-access-p2lzd\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.751535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-certs\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.751535 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-podinfo\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.751834 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-credential-socket\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.751921 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-data\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.752055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-token\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.752055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.751932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-socket\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.752379 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.752354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istiod-ca-cert\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.752465 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.752402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-certs\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.753718 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.753688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-envoy\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.754037 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.754019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-podinfo\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.760442 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.760414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-token\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.760586 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.760566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lzd\" (UniqueName: \"kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-kube-api-access-p2lzd\") pod \"e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") " pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:38.872454 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:38.872383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:39.008713 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:39.008681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:39.151940 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:39.151913    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"]
Sep 11 18:45:39.157945 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:39.157887    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:45:39.158080 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:39.158043    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:45:39.158159 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:39.158135    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:45:40.049720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:40.049677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" event={"ID":"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe","Type":"ContainerStarted","Data":"452eaf61b071b0a3d00d08cc7f3512d67c25e7d041ba2b8521db07517d4458e1"}
Sep 11 18:45:40.050174 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:40.049743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" event={"ID":"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe","Type":"ContainerStarted","Data":"9965e6c21290d7c30a6e9fa54079c5c2af4d4d333fa04af5d113e91fe2446756"}
Sep 11 18:45:40.072051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:40.071986    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" podStartSLOduration=2.071966969 podStartE2EDuration="2.071966969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:40.071428863 +0000 UTC m=+1821.168022522" watchObservedRunningTime="2026-09-11 18:45:40.071966969 +0000 UTC m=+1821.168560603"
Sep 11 18:45:40.872653 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:40.872613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:40.877998 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:40.877957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:41.055589 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:41.055550    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:41.056619 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:41.056591    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:45:41.314333 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:41.314230    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:41.969661 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:41.969612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:42.059858 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:42.059815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:42.161460 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:42.161418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:42.976522 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:42.976481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:43.945736 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:43.945695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:43.979329 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:43.979292    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"]
Sep 11 18:45:43.982520 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:43.982502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:43.984645 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:43.984609    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b0293c-serving-cert\""
Sep 11 18:45:43.992360 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:43.992326    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"]
Sep 11 18:45:44.008028 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.007995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f636f1d-8dd3-472d-822f-cab7cc6c790c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.008326 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.008137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.008326 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.008193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfwsh\" (UniqueName: \"kubernetes.io/projected/0f636f1d-8dd3-472d-822f-cab7cc6c790c-kube-api-access-xfwsh\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.008326 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.008298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-metrics-certs\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.008667 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.008645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:44.109445 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.109401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.109659 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.109457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfwsh\" (UniqueName: \"kubernetes.io/projected/0f636f1d-8dd3-472d-822f-cab7cc6c790c-kube-api-access-xfwsh\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.109659 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.109505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-metrics-certs\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.109659 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.109527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f636f1d-8dd3-472d-822f-cab7cc6c790c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.110265 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.110243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f636f1d-8dd3-472d-822f-cab7cc6c790c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.112106 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.112076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-metrics-certs\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.112248 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.112080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.117817 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.117787    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfwsh\" (UniqueName: \"kubernetes.io/projected/0f636f1d-8dd3-472d-822f-cab7cc6c790c-kube-api-access-xfwsh\") pod \"maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.148740 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.148703    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:44.294835 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.294728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:44.467120 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.467086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:44.569720 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.569692    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"]
Sep 11 18:45:44.571834 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:45:44.571798    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f636f1d_8dd3_472d_822f_cab7cc6c790c.slice/crio-fbd58adced8b18078f39518bda916907278725b7d3878249fcb050dd54d3ee1d WatchSource:0}: Error finding container fbd58adced8b18078f39518bda916907278725b7d3878249fcb050dd54d3ee1d: Status 404 returned error can't find the container with id fbd58adced8b18078f39518bda916907278725b7d3878249fcb050dd54d3ee1d
Sep 11 18:45:44.572489 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.572412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:44.934647 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:44.934606    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:45.076077 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:45.076039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv" event={"ID":"0f636f1d-8dd3-472d-822f-cab7cc6c790c","Type":"ContainerStarted","Data":"fbd58adced8b18078f39518bda916907278725b7d3878249fcb050dd54d3ee1d"}
Sep 11 18:45:45.890894 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:45.890851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:46.082592 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:46.082552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv" event={"ID":"0f636f1d-8dd3-472d-822f-cab7cc6c790c","Type":"ContainerStarted","Data":"fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad"}
Sep 11 18:45:46.083030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:46.082698    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:46.100003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:46.099947    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv" podStartSLOduration=2.4753986550000002 podStartE2EDuration="3.099928469s" totalImagesPullingTime="624.529814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:46.098532127 +0000 UTC m=+1827.195125758" watchObservedRunningTime="2026-09-11 18:45:46.099928469 +0000 UTC m=+1827.196522099"
Sep 11 18:45:51.959157 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:51.959084    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk" podUID="39b61fa4-338f-418a-83c7-9e3a6053523f" containerName="main" containerID="cri-o://9671f95d77818d85af952afc9e8d7d835cc0083e48cc00c21af817dea7bfecac" gracePeriod=45
Sep 11 18:45:52.093692 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.093659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:45:52.111822 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.111656    2580 generic.go:356] "Generic (PLEG): container finished" podID="39b61fa4-338f-418a-83c7-9e3a6053523f" containerID="9671f95d77818d85af952afc9e8d7d835cc0083e48cc00c21af817dea7bfecac" exitCode=0
Sep 11 18:45:52.111822 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.111772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk" event={"ID":"39b61fa4-338f-418a-83c7-9e3a6053523f","Type":"ContainerDied","Data":"9671f95d77818d85af952afc9e8d7d835cc0083e48cc00c21af817dea7bfecac"}
Sep 11 18:45:52.277951 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.277916    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:45:52.389948 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.389913    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-model-cache\") pod \"39b61fa4-338f-418a-83c7-9e3a6053523f\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") "
Sep 11 18:45:52.389948 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.389949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-home\") pod \"39b61fa4-338f-418a-83c7-9e3a6053523f\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") "
Sep 11 18:45:52.390193 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.390013    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-tmp-dir\") pod \"39b61fa4-338f-418a-83c7-9e3a6053523f\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") "
Sep 11 18:45:52.390193 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.390054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-dshm\") pod \"39b61fa4-338f-418a-83c7-9e3a6053523f\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") "
Sep 11 18:45:52.390193 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.390092    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7kfp\" (UniqueName: \"kubernetes.io/projected/39b61fa4-338f-418a-83c7-9e3a6053523f-kube-api-access-d7kfp\") pod \"39b61fa4-338f-418a-83c7-9e3a6053523f\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") "
Sep 11 18:45:52.390193 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.390169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b61fa4-338f-418a-83c7-9e3a6053523f-tls-certs\") pod \"39b61fa4-338f-418a-83c7-9e3a6053523f\" (UID: \"39b61fa4-338f-418a-83c7-9e3a6053523f\") "
Sep 11 18:45:52.390193 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.390166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-model-cache" (OuterVolumeSpecName: "model-cache") pod "39b61fa4-338f-418a-83c7-9e3a6053523f" (UID: "39b61fa4-338f-418a-83c7-9e3a6053523f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:52.390415 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.390269    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-home" (OuterVolumeSpecName: "home") pod "39b61fa4-338f-418a-83c7-9e3a6053523f" (UID: "39b61fa4-338f-418a-83c7-9e3a6053523f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:52.390415 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.390287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39b61fa4-338f-418a-83c7-9e3a6053523f" (UID: "39b61fa4-338f-418a-83c7-9e3a6053523f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:52.390515 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.390496    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-model-cache\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:45:52.390572 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.390520    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-home\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:45:52.390572 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.390534    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-tmp-dir\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:45:52.392332 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.392293    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b61fa4-338f-418a-83c7-9e3a6053523f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39b61fa4-338f-418a-83c7-9e3a6053523f" (UID: "39b61fa4-338f-418a-83c7-9e3a6053523f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:45:52.392457 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.392365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-dshm" (OuterVolumeSpecName: "dshm") pod "39b61fa4-338f-418a-83c7-9e3a6053523f" (UID: "39b61fa4-338f-418a-83c7-9e3a6053523f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:52.392457 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.392377    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b61fa4-338f-418a-83c7-9e3a6053523f-kube-api-access-d7kfp" (OuterVolumeSpecName: "kube-api-access-d7kfp") pod "39b61fa4-338f-418a-83c7-9e3a6053523f" (UID: "39b61fa4-338f-418a-83c7-9e3a6053523f"). InnerVolumeSpecName "kube-api-access-d7kfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:45:52.491786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.491675    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b61fa4-338f-418a-83c7-9e3a6053523f-dshm\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:45:52.491786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.491722    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7kfp\" (UniqueName: \"kubernetes.io/projected/39b61fa4-338f-418a-83c7-9e3a6053523f-kube-api-access-d7kfp\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:45:52.491786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:52.491734    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b61fa4-338f-418a-83c7-9e3a6053523f-tls-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:45:53.119287 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:53.119248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk" event={"ID":"39b61fa4-338f-418a-83c7-9e3a6053523f","Type":"ContainerDied","Data":"35d580522ecac4b48f74113c18c74d8d7304e5a0d3d450a4f4e26e73c88f873a"}
Sep 11 18:45:53.119707 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:53.119306    2580 scope.go:117] "RemoveContainer" containerID="9671f95d77818d85af952afc9e8d7d835cc0083e48cc00c21af817dea7bfecac"
Sep 11 18:45:53.119707 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:53.119343    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"
Sep 11 18:45:53.143795 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:53.143756    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"]
Sep 11 18:45:53.152892 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:53.152841    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/rate-test-model-7ca6df-kserve-65b94cfcc9-94zhk"]
Sep 11 18:45:53.395255 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:53.395183    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b61fa4-338f-418a-83c7-9e3a6053523f" path="/var/lib/kubelet/pods/39b61fa4-338f-418a-83c7-9e3a6053523f/volumes"
Sep 11 18:45:54.665586 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:54.665532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:54.767581 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:54.767535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:45:57.097452 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:57.097351    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 18:45:57.097452 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:57.097431    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 18:45:59.153169 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:59.153133    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerID="452eaf61b071b0a3d00d08cc7f3512d67c25e7d041ba2b8521db07517d4458e1" exitCode=0
Sep 11 18:45:59.153685 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:59.153194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" event={"ID":"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe","Type":"ContainerDied","Data":"452eaf61b071b0a3d00d08cc7f3512d67c25e7d041ba2b8521db07517d4458e1"}
Sep 11 18:45:59.153685 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:59.153629    2580 scope.go:117] "RemoveContainer" containerID="452eaf61b071b0a3d00d08cc7f3512d67c25e7d041ba2b8521db07517d4458e1"
Sep 11 18:45:59.154721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:59.154690    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:45:59.154796 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:59.154770    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:45:59.154834 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:45:59.154810    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:46:00.164310 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:00.164266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" event={"ID":"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe","Type":"ContainerStarted","Data":"d56fcb7526220fb5c67e048d78a57926366ea3a996aa67202e9ee8fda7f8f26b"}
Sep 11 18:46:00.873129 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:00.873079    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:46:00.873409 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:00.873383    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 11 18:46:00.873456 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:00.873441    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 11 18:46:01.873450 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:01.873404    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 11 18:46:01.873902 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:01.873474    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 11 18:46:02.305977 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:02.305862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:02.872882 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:02.872834    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 11 18:46:02.873069 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:02.872939    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 11 18:46:03.873173 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:03.873115    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 11 18:46:03.873721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:03.873217    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 11 18:46:04.873161 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:04.873112    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 11 18:46:04.873524 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:04.873487    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 11 18:46:05.032917 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:05.032875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:05.949958 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:05.949916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:46:05.950983 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:05.950938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:46:05.952621 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:05.952592    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:46:06.443579 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:06.443542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:06.954842 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:06.954811    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:07.350425 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:07.350327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:07.623228 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:07.623183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:07.911314 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:07.911214    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"]
Sep 11 18:46:07.911627 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:07.911611    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b61fa4-338f-418a-83c7-9e3a6053523f" containerName="main"
Sep 11 18:46:07.911627 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:07.911627    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b61fa4-338f-418a-83c7-9e3a6053523f" containerName="main"
Sep 11 18:46:07.911768 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:07.911697    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b61fa4-338f-418a-83c7-9e3a6053523f" containerName="main"
Sep 11 18:46:07.914815 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:07.914788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:07.917063 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:07.917036    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-946072-openshift-default-dockercfg-x7gsc\""
Sep 11 18:46:07.928753 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:07.928722    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"]
Sep 11 18:46:07.970471 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:07.970428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:08.044911 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.044867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istiod-ca-cert\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.044911 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.044912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-certs\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.045188 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.044930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-token\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.045188 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.045015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-data\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.045188 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.045103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-podinfo\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.045188 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.045148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkb76\" (UniqueName: \"kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-kube-api-access-pkb76\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.045188 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.045174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-socket\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.045468 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.045264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-envoy\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.045468 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.045301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-credential-socket\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.063101 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.063060    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:08.146385 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkb76\" (UniqueName: \"kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-kube-api-access-pkb76\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.146385 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-socket\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.146662 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-envoy\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.146662 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-credential-socket\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.146662 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istiod-ca-cert\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.146662 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-certs\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.146904 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-token\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.146904 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-data\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.146904 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-podinfo\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.146904 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-socket\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.147731 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.147671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-data\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.147731 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.146944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-credential-socket\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.147946 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.147893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istiod-ca-cert\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.151224 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.148276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-certs\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.157168 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.157120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-envoy\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.158500 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.158465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-token\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.158774 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.158736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-podinfo\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.158851 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.158828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkb76\" (UniqueName: \"kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-kube-api-access-pkb76\") pod \"e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") " pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.226241 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.226128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:08.249940 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.249841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:08.506984 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.506893    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"]
Sep 11 18:46:08.513712 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:46:08.513644    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8148e2_33d9_454e_a4e9_4dc11caa88c9.slice/crio-60794299e4254995352010c7a2259e7c6baf753af92ee803d1a764f57e00fce6 WatchSource:0}: Error finding container 60794299e4254995352010c7a2259e7c6baf753af92ee803d1a764f57e00fce6: Status 404 returned error can't find the container with id 60794299e4254995352010c7a2259e7c6baf753af92ee803d1a764f57e00fce6
Sep 11 18:46:08.515601 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.515562    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:46:08.515710 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.515637    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:46:08.515710 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:08.515666    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:46:09.013961 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:09.013918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:09.214448 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:09.214406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" event={"ID":"bd8148e2-33d9-454e-a4e9-4dc11caa88c9","Type":"ContainerStarted","Data":"1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2"}
Sep 11 18:46:09.214639 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:09.214465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" event={"ID":"bd8148e2-33d9-454e-a4e9-4dc11caa88c9","Type":"ContainerStarted","Data":"60794299e4254995352010c7a2259e7c6baf753af92ee803d1a764f57e00fce6"}
Sep 11 18:46:09.226463 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:09.226428    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:09.231678 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:09.231643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:09.233308 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:09.233257    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" podStartSLOduration=2.233239737 podStartE2EDuration="2.233239737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:46:09.232458924 +0000 UTC m=+1850.329052556" watchObservedRunningTime="2026-09-11 18:46:09.233239737 +0000 UTC m=+1850.329833367"
Sep 11 18:46:10.219277 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:10.219228    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:10.220194 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:10.220172    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:10.351120 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:10.351077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:11.145827 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:11.145789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:11.248156 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:11.248112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:12.132783 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:12.132745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:12.855978 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:12.855936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:13.639507 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:13.639458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:14.806549 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:14.806485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:30.318211 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:30.318174    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerID="1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2" exitCode=0
Sep 11 18:46:30.318668 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:30.318257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" event={"ID":"bd8148e2-33d9-454e-a4e9-4dc11caa88c9","Type":"ContainerDied","Data":"1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2"}
Sep 11 18:46:30.318721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:30.318679    2580 scope.go:117] "RemoveContainer" containerID="1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2"
Sep 11 18:46:30.319790 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:30.319758    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:46:30.319858 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:30.319830    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:46:30.319899 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:30.319861    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 18:46:31.325340 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:31.325303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" event={"ID":"bd8148e2-33d9-454e-a4e9-4dc11caa88c9","Type":"ContainerStarted","Data":"230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927"}
Sep 11 18:46:32.226980 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:32.226925    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:32.227178 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:32.227157    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 11 18:46:32.227257 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:32.227235    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 11 18:46:33.226843 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:33.226801    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 11 18:46:33.227267 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:33.226864    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 11 18:46:34.227224 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:34.227145    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 11 18:46:34.227724 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:34.227248    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 11 18:46:34.438098 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:34.438017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:35.227155 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:35.227106    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 11 18:46:35.227360 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:35.227174    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 11 18:46:36.110937 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:36.110898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:36.215235 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:36.215177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:36.227361 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:36.227319    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 11 18:46:36.227746 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:36.227381    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 11 18:46:36.974841 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:36.974801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:37.231192 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:37.231101    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:37.231571 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:37.231481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:37.232349 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:37.232331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:46:37.716785 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:37.716743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:38.539648 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:38.539602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:39.392319 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:39.392283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:46:48.565092 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:46:48.565006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:12.348866 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:12.348822    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:12.645766 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:12.645716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:12.783132 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:12.783095    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:14.855732 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:14.855680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:15.635548 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:15.635502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:15.736305 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:15.736253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:15.835216 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:15.835169    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:16.706301 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:16.706261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:17.643927 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:17.643874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:18.500626 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:18.500580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:19.357418 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.357377    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"]
Sep 11 18:47:19.360993 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.360966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.363082 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.363056    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-e9c09a-serving-cert\""
Sep 11 18:47:19.371859 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.371826    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"]
Sep 11 18:47:19.403414 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.403379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:19.413682 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.413642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.413859 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.413776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc2xc\" (UniqueName: \"kubernetes.io/projected/9d678cf4-0f5d-41c4-b19f-535da8abbf83-kube-api-access-bc2xc\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.413859 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.413823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d678cf4-0f5d-41c4-b19f-535da8abbf83-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.414042 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.414009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.506657 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.506619    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:19.515171 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.515134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.515171 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.515175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.515447 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.515238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc2xc\" (UniqueName: \"kubernetes.io/projected/9d678cf4-0f5d-41c4-b19f-535da8abbf83-kube-api-access-bc2xc\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.515447 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.515268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d678cf4-0f5d-41c4-b19f-535da8abbf83-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.515447 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:47:19.515362    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-serving-cert: secret "maas-api-e2e-ipp-cleanup-e9c09a-serving-cert" not found
Sep 11 18:47:19.515447 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:47:19.515439    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-maas-api-tls podName:9d678cf4-0f5d-41c4-b19f-535da8abbf83 nodeName:}" failed. No retries permitted until 2026-09-11 18:47:20.015416947 +0000 UTC m=+1921.112010563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx" (UID: "9d678cf4-0f5d-41c4-b19f-535da8abbf83") : secret "maas-api-e2e-ipp-cleanup-e9c09a-serving-cert" not found
Sep 11 18:47:19.515951 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.515924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d678cf4-0f5d-41c4-b19f-535da8abbf83-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.517801 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.517778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.526587 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.526555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc2xc\" (UniqueName: \"kubernetes.io/projected/9d678cf4-0f5d-41c4-b19f-535da8abbf83-kube-api-access-bc2xc\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:19.793046 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.793000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:19.912147 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:19.912112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:20.014498 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:20.014445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:20.019567 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:20.019530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:20.022138 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:20.022110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:20.274500 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:20.274448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:20.556042 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:20.556014    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"]
Sep 11 18:47:20.558076 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:47:20.558031    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d678cf4_0f5d_41c4_b19f_535da8abbf83.slice/crio-1bc78c391044b13a03e03fa90357ea930dedcb8a952680ec6cb6bdcc73bd80d4 WatchSource:0}: Error finding container 1bc78c391044b13a03e03fa90357ea930dedcb8a952680ec6cb6bdcc73bd80d4: Status 404 returned error can't find the container with id 1bc78c391044b13a03e03fa90357ea930dedcb8a952680ec6cb6bdcc73bd80d4
Sep 11 18:47:20.566618 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:20.566577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx" event={"ID":"9d678cf4-0f5d-41c4-b19f-535da8abbf83","Type":"ContainerStarted","Data":"1bc78c391044b13a03e03fa90357ea930dedcb8a952680ec6cb6bdcc73bd80d4"}
Sep 11 18:47:21.572891 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:21.572843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx" event={"ID":"9d678cf4-0f5d-41c4-b19f-535da8abbf83","Type":"ContainerStarted","Data":"27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539"}
Sep 11 18:47:21.573411 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:21.572949    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:21.590529 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:21.590464    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx" podStartSLOduration=2.201033256 podStartE2EDuration="2.590443523s" totalImagesPullingTime="389.410267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:47:21.588568589 +0000 UTC m=+1922.685162220" watchObservedRunningTime="2026-09-11 18:47:21.590443523 +0000 UTC m=+1922.687037154"
Sep 11 18:47:27.583343 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:27.583311    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:37.012008 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:37.011966    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:41.532707 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:41.532664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:42.296036 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:42.295983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:43.149990 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:43.149950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:43.864965 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:43.864908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:44.711728 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:44.711675    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:45.636945 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:45.636906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:45.735700 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:45.735652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:47.097756 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:47.097718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:47.926783 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:47.926736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:48.749281 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:48.749231    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:49.475794 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:49.475751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:50.236221 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:50.236155    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:51.044682 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:51.044642    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:57.468473 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.468430    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"]
Sep 11 18:47:57.468980 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.468709    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx" podUID="9d678cf4-0f5d-41c4-b19f-535da8abbf83" containerName="maas-api" containerID="cri-o://27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539" gracePeriod=30
Sep 11 18:47:57.496873 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.496829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:57.572404 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.572368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:57.578396 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.578346    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx" podUID="9d678cf4-0f5d-41c4-b19f-535da8abbf83" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": dial tcp 10.133.0.59:8443: connect: connection refused"
Sep 11 18:47:57.671024 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.670982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:57.714313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.714283    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:57.742903 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.742756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:57.753902 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.753865    2580 generic.go:356] "Generic (PLEG): container finished" podID="9d678cf4-0f5d-41c4-b19f-535da8abbf83" containerID="27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539" exitCode=0
Sep 11 18:47:57.754117 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.753927    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"
Sep 11 18:47:57.754117 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.753977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx" event={"ID":"9d678cf4-0f5d-41c4-b19f-535da8abbf83","Type":"ContainerDied","Data":"27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539"}
Sep 11 18:47:57.754117 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.754057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx" event={"ID":"9d678cf4-0f5d-41c4-b19f-535da8abbf83","Type":"ContainerDied","Data":"1bc78c391044b13a03e03fa90357ea930dedcb8a952680ec6cb6bdcc73bd80d4"}
Sep 11 18:47:57.754117 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.754094    2580 scope.go:117] "RemoveContainer" containerID="27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539"
Sep 11 18:47:57.764017 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.763993    2580 scope.go:117] "RemoveContainer" containerID="27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539"
Sep 11 18:47:57.764365 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:47:57.764344    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539\": container with ID starting with 27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539 not found: ID does not exist" containerID="27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539"
Sep 11 18:47:57.764485 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.764373    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539"} err="failed to get container status \"27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539\": rpc error: code = NotFound desc = could not find container \"27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539\": container with ID starting with 27a0168281c139d4a9615e773826a05d601a455df671f3e2cb82170d1ea6d539 not found: ID does not exist"
Sep 11 18:47:57.886230 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.886183    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-metrics-certs\") pod \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") "
Sep 11 18:47:57.886419 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.886274    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d678cf4-0f5d-41c4-b19f-535da8abbf83-openshift-service-ca\") pod \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") "
Sep 11 18:47:57.886419 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.886353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-maas-api-tls\") pod \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") "
Sep 11 18:47:57.886419 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.886407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc2xc\" (UniqueName: \"kubernetes.io/projected/9d678cf4-0f5d-41c4-b19f-535da8abbf83-kube-api-access-bc2xc\") pod \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\" (UID: \"9d678cf4-0f5d-41c4-b19f-535da8abbf83\") "
Sep 11 18:47:57.886670 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.886644    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d678cf4-0f5d-41c4-b19f-535da8abbf83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d678cf4-0f5d-41c4-b19f-535da8abbf83" (UID: "9d678cf4-0f5d-41c4-b19f-535da8abbf83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:47:57.888636 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.888599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d678cf4-0f5d-41c4-b19f-535da8abbf83" (UID: "9d678cf4-0f5d-41c4-b19f-535da8abbf83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:47:57.888798 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.888630    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9d678cf4-0f5d-41c4-b19f-535da8abbf83" (UID: "9d678cf4-0f5d-41c4-b19f-535da8abbf83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:47:57.888798 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.888652    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d678cf4-0f5d-41c4-b19f-535da8abbf83-kube-api-access-bc2xc" (OuterVolumeSpecName: "kube-api-access-bc2xc") pod "9d678cf4-0f5d-41c4-b19f-535da8abbf83" (UID: "9d678cf4-0f5d-41c4-b19f-535da8abbf83"). InnerVolumeSpecName "kube-api-access-bc2xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:47:57.988178 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.988132    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:47:57.988178 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.988171    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc2xc\" (UniqueName: \"kubernetes.io/projected/9d678cf4-0f5d-41c4-b19f-535da8abbf83-kube-api-access-bc2xc\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:47:57.988178 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.988181    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d678cf4-0f5d-41c4-b19f-535da8abbf83-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:47:57.988178 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:57.988190    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d678cf4-0f5d-41c4-b19f-535da8abbf83-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:47:58.077638 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:58.077598    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"]
Sep 11 18:47:58.082122 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:58.082085    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e9c09a-b6c576457-pgbrx"]
Sep 11 18:47:58.098602 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:58.098565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:58.215525 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:58.215481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:58.938932 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:58.938890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:47:59.396261 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:59.396225    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d678cf4-0f5d-41c4-b19f-535da8abbf83" path="/var/lib/kubelet/pods/9d678cf4-0f5d-41c4-b19f-535da8abbf83/volumes"
Sep 11 18:47:59.961211 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:47:59.961143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:00.705055 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:00.705008    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:01.427070 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:01.427029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:04.600891 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:04.600845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:04.671224 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:04.671178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:04.869657 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:04.869569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:08.537456 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:08.537402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:09.331398 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:09.331358    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:10.235450 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:10.235409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:10.958411 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:10.958368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:11.675106 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:11.675054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:12.539760 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:12.539723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:23.911463 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:23.911424    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"]
Sep 11 18:48:23.911971 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:23.911729    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv" podUID="0f636f1d-8dd3-472d-822f-cab7cc6c790c" containerName="maas-api" containerID="cri-o://fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad" gracePeriod=30
Sep 11 18:48:23.939584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:23.939544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:24.014374 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.014337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:24.073551 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.073516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:24.162501 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.162410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:24.169118 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.169087    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:48:24.221165 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.221132    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:24.237415 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.237380    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-maas-api-tls\") pod \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") "
Sep 11 18:48:24.237621 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.237451    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f636f1d-8dd3-472d-822f-cab7cc6c790c-openshift-service-ca\") pod \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") "
Sep 11 18:48:24.237621 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.237475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfwsh\" (UniqueName: \"kubernetes.io/projected/0f636f1d-8dd3-472d-822f-cab7cc6c790c-kube-api-access-xfwsh\") pod \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") "
Sep 11 18:48:24.237621 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.237527    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-metrics-certs\") pod \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\" (UID: \"0f636f1d-8dd3-472d-822f-cab7cc6c790c\") "
Sep 11 18:48:24.237896 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.237866    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f636f1d-8dd3-472d-822f-cab7cc6c790c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f636f1d-8dd3-472d-822f-cab7cc6c790c" (UID: "0f636f1d-8dd3-472d-822f-cab7cc6c790c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:48:24.239819 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.239787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0f636f1d-8dd3-472d-822f-cab7cc6c790c" (UID: "0f636f1d-8dd3-472d-822f-cab7cc6c790c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:48:24.239819 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.239808    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f636f1d-8dd3-472d-822f-cab7cc6c790c" (UID: "0f636f1d-8dd3-472d-822f-cab7cc6c790c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:48:24.240019 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.239840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f636f1d-8dd3-472d-822f-cab7cc6c790c-kube-api-access-xfwsh" (OuterVolumeSpecName: "kube-api-access-xfwsh") pod "0f636f1d-8dd3-472d-822f-cab7cc6c790c" (UID: "0f636f1d-8dd3-472d-822f-cab7cc6c790c"). InnerVolumeSpecName "kube-api-access-xfwsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:48:24.339270 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.339231    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f636f1d-8dd3-472d-822f-cab7cc6c790c-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:48:24.339270 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.339263    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfwsh\" (UniqueName: \"kubernetes.io/projected/0f636f1d-8dd3-472d-822f-cab7cc6c790c-kube-api-access-xfwsh\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:48:24.339270 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.339275    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:48:24.339556 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.339285    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f636f1d-8dd3-472d-822f-cab7cc6c790c-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:48:24.544880 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.544792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:24.649218 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.649170    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:24.893624 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.893586    2580 generic.go:356] "Generic (PLEG): container finished" podID="0f636f1d-8dd3-472d-822f-cab7cc6c790c" containerID="fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad" exitCode=0
Sep 11 18:48:24.893839 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.893632    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"
Sep 11 18:48:24.893839 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.893646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv" event={"ID":"0f636f1d-8dd3-472d-822f-cab7cc6c790c","Type":"ContainerDied","Data":"fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad"}
Sep 11 18:48:24.893839 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.893697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv" event={"ID":"0f636f1d-8dd3-472d-822f-cab7cc6c790c","Type":"ContainerDied","Data":"fbd58adced8b18078f39518bda916907278725b7d3878249fcb050dd54d3ee1d"}
Sep 11 18:48:24.893839 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.893723    2580 scope.go:117] "RemoveContainer" containerID="fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad"
Sep 11 18:48:24.903096 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.903076    2580 scope.go:117] "RemoveContainer" containerID="fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad"
Sep 11 18:48:24.903430 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:48:24.903408    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad\": container with ID starting with fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad not found: ID does not exist" containerID="fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad"
Sep 11 18:48:24.903516 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.903439    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad"} err="failed to get container status \"fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad\": rpc error: code = NotFound desc = could not find container \"fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad\": container with ID starting with fe97d487ea33168abd5f8964ce9bc7db827a9da5bfc55bbb97251d5b2c44f2ad not found: ID does not exist"
Sep 11 18:48:24.915141 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.915108    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"]
Sep 11 18:48:24.919644 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:24.919615    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b0293c-564766fb96-xlsbv"]
Sep 11 18:48:25.396330 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:25.396295    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f636f1d-8dd3-472d-822f-cab7cc6c790c" path="/var/lib/kubelet/pods/0f636f1d-8dd3-472d-822f-cab7cc6c790c/volumes"
Sep 11 18:48:25.436169 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:25.436134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:26.301747 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:26.301706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:27.054563 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:27.054525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:29.567582 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:29.567538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:40.991737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:40.991687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:48:41.054693 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:48:41.054641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:06.542563 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:06.542515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:07.112883 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:07.112851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:12.844857 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:12.844815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:18.677220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:18.677151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:18.779952 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:18.779912    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:18.931368 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:18.931265    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"]
Sep 11 18:49:18.931547 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:18.931489    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy" containerID="cri-o://d56fcb7526220fb5c67e048d78a57926366ea3a996aa67202e9ee8fda7f8f26b" gracePeriod=30
Sep 11 18:49:18.955432 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:18.955388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:19.626046 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:19.626007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:20.483793 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:20.483736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:20.952662 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:20.952618    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:49:20.952833 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:20.952699    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:49:21.205007 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:21.204910    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:22.099034 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:22.098985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:22.850861 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:22.850818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:24.184364 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.184332    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerID="d56fcb7526220fb5c67e048d78a57926366ea3a996aa67202e9ee8fda7f8f26b" exitCode=0
Sep 11 18:49:24.184823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.184393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" event={"ID":"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe","Type":"ContainerDied","Data":"d56fcb7526220fb5c67e048d78a57926366ea3a996aa67202e9ee8fda7f8f26b"}
Sep 11 18:49:24.184823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.184443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2" event={"ID":"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe","Type":"ContainerDied","Data":"9965e6c21290d7c30a6e9fa54079c5c2af4d4d333fa04af5d113e91fe2446756"}
Sep 11 18:49:24.184823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.184463    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9965e6c21290d7c30a6e9fa54079c5c2af4d4d333fa04af5d113e91fe2446756"
Sep 11 18:49:24.184823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.184480    2580 scope.go:117] "RemoveContainer" containerID="452eaf61b071b0a3d00d08cc7f3512d67c25e7d041ba2b8521db07517d4458e1"
Sep 11 18:49:24.197274 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.197251    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:49:24.301165 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301128    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-data\") pod \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") "
Sep 11 18:49:24.301381 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301227    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-envoy\") pod \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") "
Sep 11 18:49:24.301381 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301254    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-credential-socket\") pod \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") "
Sep 11 18:49:24.301381 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-podinfo\") pod \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") "
Sep 11 18:49:24.301381 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-socket\") pod \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") "
Sep 11 18:49:24.301381 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301344    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-certs\") pod \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") "
Sep 11 18:49:24.301381 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301367    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istiod-ca-cert\") pod \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") "
Sep 11 18:49:24.301694 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301402    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2lzd\" (UniqueName: \"kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-kube-api-access-p2lzd\") pod \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") "
Sep 11 18:49:24.301694 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301426    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-token\") pod \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\" (UID: \"8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe\") "
Sep 11 18:49:24.301694 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301518    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" (UID: "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:49:24.301694 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" (UID: "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:49:24.301903 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301774    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-credential-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:24.301903 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301793    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-data\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:24.301903 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301803    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" (UID: "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:49:24.301903 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" (UID: "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:49:24.301903 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.301864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" (UID: "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:49:24.303571 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.303545    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" (UID: "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:49:24.303721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.303570    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" (UID: "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:49:24.303721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.303616    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" (UID: "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:49:24.303721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.303671    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-kube-api-access-p2lzd" (OuterVolumeSpecName: "kube-api-access-p2lzd") pod "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" (UID: "8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe"). InnerVolumeSpecName "kube-api-access-p2lzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:49:24.402413 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.402364    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:24.402413 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.402399    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istiod-ca-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:24.402413 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.402408    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2lzd\" (UniqueName: \"kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-kube-api-access-p2lzd\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:24.402413 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.402419    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-token\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:24.402413 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.402428    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-envoy\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:24.402722 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.402435    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-istio-podinfo\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:24.402722 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:24.402442    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe-workload-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:25.190438 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:25.190406    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"
Sep 11 18:49:25.223657 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:25.223609    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"]
Sep 11 18:49:25.228250 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:25.228209    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b0293c-openshift-default-585dcc9f6-xwtw2"]
Sep 11 18:49:25.395926 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:25.395890    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" path="/var/lib/kubelet/pods/8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe/volumes"
Sep 11 18:49:29.870974 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:29.870932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:29.942558 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:29.942506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:30.001011 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:30.000964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:30.087932 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:30.087893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:30.149600 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:30.149556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:30.472018 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:30.471931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:30.568802 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:30.568744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:31.343134 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:31.343094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:32.253689 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:32.253652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:36.652442 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:36.652404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:36.753157 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:36.753117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:36.905649 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:36.905558    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"]
Sep 11 18:49:36.905854 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:36.905821    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy" containerID="cri-o://230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927" gracePeriod=30
Sep 11 18:49:36.936997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:36.936952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:37.234400 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:37.234311    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:49:37.234400 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:37.234373    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:49:37.430457 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:37.430419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:38.332664 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:38.332626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:39.320040 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:39.319995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:40.164001 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:40.163947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:42.163878 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.163847    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:49:42.262373 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkb76\" (UniqueName: \"kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-kube-api-access-pkb76\") pod \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") "
Sep 11 18:49:42.262373 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-credential-socket\") pod \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") "
Sep 11 18:49:42.262373 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262363    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-certs\") pod \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") "
Sep 11 18:49:42.262681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262392    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-token\") pod \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") "
Sep 11 18:49:42.262681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-podinfo\") pod \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") "
Sep 11 18:49:42.262681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262491    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-envoy\") pod \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") "
Sep 11 18:49:42.262681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istiod-ca-cert\") pod \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") "
Sep 11 18:49:42.262681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262565    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-socket\") pod \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") "
Sep 11 18:49:42.262681 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-data\") pod \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\" (UID: \"bd8148e2-33d9-454e-a4e9-4dc11caa88c9\") "
Sep 11 18:49:42.262997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd8148e2-33d9-454e-a4e9-4dc11caa88c9" (UID: "bd8148e2-33d9-454e-a4e9-4dc11caa88c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:49:42.262997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd8148e2-33d9-454e-a4e9-4dc11caa88c9" (UID: "bd8148e2-33d9-454e-a4e9-4dc11caa88c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:49:42.262997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262932    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-credential-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:42.262997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262954    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:42.262997 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.262932    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd8148e2-33d9-454e-a4e9-4dc11caa88c9" (UID: "bd8148e2-33d9-454e-a4e9-4dc11caa88c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:49:42.263276 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.263056    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd8148e2-33d9-454e-a4e9-4dc11caa88c9" (UID: "bd8148e2-33d9-454e-a4e9-4dc11caa88c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:49:42.263276 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.263188    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd8148e2-33d9-454e-a4e9-4dc11caa88c9" (UID: "bd8148e2-33d9-454e-a4e9-4dc11caa88c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:49:42.265120 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.265077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd8148e2-33d9-454e-a4e9-4dc11caa88c9" (UID: "bd8148e2-33d9-454e-a4e9-4dc11caa88c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:49:42.265120 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.265103    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd8148e2-33d9-454e-a4e9-4dc11caa88c9" (UID: "bd8148e2-33d9-454e-a4e9-4dc11caa88c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:49:42.265312 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.265152    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd8148e2-33d9-454e-a4e9-4dc11caa88c9" (UID: "bd8148e2-33d9-454e-a4e9-4dc11caa88c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:49:42.265312 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.265187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-kube-api-access-pkb76" (OuterVolumeSpecName: "kube-api-access-pkb76") pod "bd8148e2-33d9-454e-a4e9-4dc11caa88c9" (UID: "bd8148e2-33d9-454e-a4e9-4dc11caa88c9"). InnerVolumeSpecName "kube-api-access-pkb76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:49:42.272816 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.272781    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerID="230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927" exitCode=0
Sep 11 18:49:42.272948 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.272836    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"
Sep 11 18:49:42.272948 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.272855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" event={"ID":"bd8148e2-33d9-454e-a4e9-4dc11caa88c9","Type":"ContainerDied","Data":"230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927"}
Sep 11 18:49:42.272948 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.272914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt" event={"ID":"bd8148e2-33d9-454e-a4e9-4dc11caa88c9","Type":"ContainerDied","Data":"60794299e4254995352010c7a2259e7c6baf753af92ee803d1a764f57e00fce6"}
Sep 11 18:49:42.273103 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.272974    2580 scope.go:117] "RemoveContainer" containerID="230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927"
Sep 11 18:49:42.284641 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.284620    2580 scope.go:117] "RemoveContainer" containerID="1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2"
Sep 11 18:49:42.292880 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.292856    2580 scope.go:117] "RemoveContainer" containerID="230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927"
Sep 11 18:49:42.293180 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:49:42.293157    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927\": container with ID starting with 230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927 not found: ID does not exist" containerID="230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927"
Sep 11 18:49:42.293251 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.293189    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927"} err="failed to get container status \"230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927\": rpc error: code = NotFound desc = could not find container \"230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927\": container with ID starting with 230906926a430b63957250011e4f456027a6aa4eb4b9057921332e84d11dd927 not found: ID does not exist"
Sep 11 18:49:42.293251 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.293230    2580 scope.go:117] "RemoveContainer" containerID="1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2"
Sep 11 18:49:42.293530 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:49:42.293508    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2\": container with ID starting with 1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2 not found: ID does not exist" containerID="1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2"
Sep 11 18:49:42.293591 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.293536    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2"} err="failed to get container status \"1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2\": rpc error: code = NotFound desc = could not find container \"1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2\": container with ID starting with 1efb8453f8253e23f12db830919e4199bad3eadb6e2ceb20f9c874d1d89d9bc2 not found: ID does not exist"
Sep 11 18:49:42.301613 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.301582    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"]
Sep 11 18:49:42.305991 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.305961    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-946072-openshift-default-59bc5bdc96-zr9wt"]
Sep 11 18:49:42.363455 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.363424    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-data\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:42.363455 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.363451    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkb76\" (UniqueName: \"kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-kube-api-access-pkb76\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:42.363455 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.363462    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-token\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:42.363695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.363473    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-podinfo\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:42.363695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.363481    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istio-envoy\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:42.363695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.363491    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-istiod-ca-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:42.363695 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:42.363499    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8148e2-33d9-454e-a4e9-4dc11caa88c9-workload-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:49:43.394986 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:43.394951    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" path="/var/lib/kubelet/pods/bd8148e2-33d9-454e-a4e9-4dc11caa88c9/volumes"
Sep 11 18:49:47.884745 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:47.884709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:47.957604 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:47.957565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:48.019798 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:48.019755    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:48.117702 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:48.117656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:48.205011 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:48.204908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:48.509471 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:48.509372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:48.585554 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:48.585518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:49.415655 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:49.415613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:55.004332 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:55.004292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:55.066610 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:55.066569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:55.272152 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:55.272054    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"]
Sep 11 18:49:55.272355 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:55.272304    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy" containerID="cri-o://6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8" gracePeriod=30
Sep 11 18:49:55.285307 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:55.285267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:56.537046 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:56.536985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:49:57.546107 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:49:57.546070    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:00.532585 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.532553    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:50:00.626527 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.626494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-envoy\") pod \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") "
Sep 11 18:50:00.626721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.626539    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-podinfo\") pod \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") "
Sep 11 18:50:00.626721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.626583    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-token\") pod \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") "
Sep 11 18:50:00.626721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.626608    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfhrm\" (UniqueName: \"kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-kube-api-access-jfhrm\") pod \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") "
Sep 11 18:50:00.626721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.626637    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-certs\") pod \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") "
Sep 11 18:50:00.626721 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.626673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-credential-socket\") pod \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") "
Sep 11 18:50:00.627000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.626758    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-socket\") pod \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") "
Sep 11 18:50:00.627000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.626804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istiod-ca-cert\") pod \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") "
Sep 11 18:50:00.627000 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.626843    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-data\") pod \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\" (UID: \"bbda8b7d-4c3b-498f-bc1a-5995a49c41db\") "
Sep 11 18:50:00.627159 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.626997    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbda8b7d-4c3b-498f-bc1a-5995a49c41db" (UID: "bbda8b7d-4c3b-498f-bc1a-5995a49c41db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:00.627159 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.627022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbda8b7d-4c3b-498f-bc1a-5995a49c41db" (UID: "bbda8b7d-4c3b-498f-bc1a-5995a49c41db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:00.627293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.627225    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.627293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.627249    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-credential-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.627293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.627258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbda8b7d-4c3b-498f-bc1a-5995a49c41db" (UID: "bbda8b7d-4c3b-498f-bc1a-5995a49c41db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:00.627459 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.627327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbda8b7d-4c3b-498f-bc1a-5995a49c41db" (UID: "bbda8b7d-4c3b-498f-bc1a-5995a49c41db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:00.627575 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.627551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbda8b7d-4c3b-498f-bc1a-5995a49c41db" (UID: "bbda8b7d-4c3b-498f-bc1a-5995a49c41db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:50:00.628955 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.628924    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbda8b7d-4c3b-498f-bc1a-5995a49c41db" (UID: "bbda8b7d-4c3b-498f-bc1a-5995a49c41db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:00.628955 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.628943    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-kube-api-access-jfhrm" (OuterVolumeSpecName: "kube-api-access-jfhrm") pod "bbda8b7d-4c3b-498f-bc1a-5995a49c41db" (UID: "bbda8b7d-4c3b-498f-bc1a-5995a49c41db"). InnerVolumeSpecName "kube-api-access-jfhrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:00.629099 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.628968    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbda8b7d-4c3b-498f-bc1a-5995a49c41db" (UID: "bbda8b7d-4c3b-498f-bc1a-5995a49c41db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:00.629099 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.629001    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbda8b7d-4c3b-498f-bc1a-5995a49c41db" (UID: "bbda8b7d-4c3b-498f-bc1a-5995a49c41db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:50:00.728386 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.728346    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-workload-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.728386 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.728377    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istiod-ca-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.728386 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.728387    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-data\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.728386 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.728395    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-envoy\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.728744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.728403    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-podinfo\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.728744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.728410    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-istio-token\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.728744 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:00.728418    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfhrm\" (UniqueName: \"kubernetes.io/projected/bbda8b7d-4c3b-498f-bc1a-5995a49c41db-kube-api-access-jfhrm\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:01.366005 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.365965    2580 generic.go:356] "Generic (PLEG): container finished" podID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerID="6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8" exitCode=0
Sep 11 18:50:01.366226 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.366019    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"
Sep 11 18:50:01.366226 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.366049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" event={"ID":"bbda8b7d-4c3b-498f-bc1a-5995a49c41db","Type":"ContainerDied","Data":"6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8"}
Sep 11 18:50:01.366226 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.366144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g" event={"ID":"bbda8b7d-4c3b-498f-bc1a-5995a49c41db","Type":"ContainerDied","Data":"1533c6320331e1b0f767ebe55ae1bdca982c1fd3f42b1cfdd95d562d8824a183"}
Sep 11 18:50:01.366226 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.366168    2580 scope.go:117] "RemoveContainer" containerID="6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8"
Sep 11 18:50:01.375415 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.375390    2580 scope.go:117] "RemoveContainer" containerID="aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274"
Sep 11 18:50:01.384189 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.384162    2580 scope.go:117] "RemoveContainer" containerID="6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8"
Sep 11 18:50:01.384656 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:50:01.384630    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8\": container with ID starting with 6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8 not found: ID does not exist" containerID="6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8"
Sep 11 18:50:01.384764 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.384665    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8"} err="failed to get container status \"6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8\": rpc error: code = NotFound desc = could not find container \"6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8\": container with ID starting with 6fb83e0dfa1f8da5ceeddedaa38530929bf574a0a5e9be4915dc14925cd6c7d8 not found: ID does not exist"
Sep 11 18:50:01.384764 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.384688    2580 scope.go:117] "RemoveContainer" containerID="aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274"
Sep 11 18:50:01.385004 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:50:01.384984    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274\": container with ID starting with aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274 not found: ID does not exist" containerID="aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274"
Sep 11 18:50:01.385054 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.385011    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274"} err="failed to get container status \"aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274\": rpc error: code = NotFound desc = could not find container \"aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274\": container with ID starting with aa343a5f6f2f72895cc43d5f8a0b0931288f87111cf0f0d9e12f323710390274 not found: ID does not exist"
Sep 11 18:50:01.389046 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.389015    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"]
Sep 11 18:50:01.396677 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:01.396648    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7ca6df-openshift-default-7496855878-kcb4g"]
Sep 11 18:50:03.394595 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:03.394558    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" path="/var/lib/kubelet/pods/bbda8b7d-4c3b-498f-bc1a-5995a49c41db/volumes"
Sep 11 18:50:06.146293 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:06.146240    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:06.158093 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:06.158058    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q"]
Sep 11 18:50:06.161739 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:06.161706    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-711a34-29819205-trb5q"]
Sep 11 18:50:06.239170 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:06.239100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:06.299968 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:06.299930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:06.382329 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:06.382295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:06.445560 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:06.445457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:06.760275 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:06.760132    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:06.860182 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:06.860141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:07.396503 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:07.396467    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32e14f13-ba29-4ad0-86c6-fda8abbf4b43" path="/var/lib/kubelet/pods/32e14f13-ba29-4ad0-86c6-fda8abbf4b43/volumes"
Sep 11 18:50:13.703078 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:13.703045    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:13.802564 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:13.802526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:13.934616 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:13.934582    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"]
Sep 11 18:50:13.934847 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:13.934821    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy" containerID="cri-o://4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f" gracePeriod=30
Sep 11 18:50:13.963917 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:13.963818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:14.496523 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:14.496482    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:50:14.496700 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:14.496546    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:50:14.881733 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:14.881689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:16.700448 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:16.700410    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dd795d9-bqnzv"]
Sep 11 18:50:16.700958 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:16.700656    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6dd795d9-bqnzv" podUID="5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f" containerName="manager" containerID="cri-o://2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5" gracePeriod=30
Sep 11 18:50:16.937125 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:16.937097    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:50:16.965291 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:16.965178    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-webhook-cert\") pod \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") "
Sep 11 18:50:16.965291 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:16.965265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxl8p\" (UniqueName: \"kubernetes.io/projected/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-kube-api-access-wxl8p\") pod \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") "
Sep 11 18:50:16.965525 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:16.965336    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-metrics-certs\") pod \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\" (UID: \"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f\") "
Sep 11 18:50:16.967699 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:16.967630    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f" (UID: "5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:50:16.967852 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:16.967789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f" (UID: "5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:50:16.968069 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:16.968039    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-kube-api-access-wxl8p" (OuterVolumeSpecName: "kube-api-access-wxl8p") pod "5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f" (UID: "5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f"). InnerVolumeSpecName "kube-api-access-wxl8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:17.066584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.066530    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-webhook-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:17.066584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.066581    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxl8p\" (UniqueName: \"kubernetes.io/projected/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-kube-api-access-wxl8p\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:17.066584 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.066592    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:17.443653 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.443615    2580 generic.go:356] "Generic (PLEG): container finished" podID="5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f" containerID="2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5" exitCode=0
Sep 11 18:50:17.443823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.443665    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd795d9-bqnzv"
Sep 11 18:50:17.443823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.443679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd795d9-bqnzv" event={"ID":"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f","Type":"ContainerDied","Data":"2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5"}
Sep 11 18:50:17.443823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.443721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd795d9-bqnzv" event={"ID":"5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f","Type":"ContainerDied","Data":"85c5d165cb617a13f3e57676173df7c7118009bcddc804c48f9df26d3b3d03c1"}
Sep 11 18:50:17.443823 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.443742    2580 scope.go:117] "RemoveContainer" containerID="2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5"
Sep 11 18:50:17.452884 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.452861    2580 scope.go:117] "RemoveContainer" containerID="2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5"
Sep 11 18:50:17.453217 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:50:17.453179    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5\": container with ID starting with 2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5 not found: ID does not exist" containerID="2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5"
Sep 11 18:50:17.453309 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.453228    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5"} err="failed to get container status \"2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5\": rpc error: code = NotFound desc = could not find container \"2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5\": container with ID starting with 2c589a9279a2349ac685b093d623c0837e12cfe4046aa5fc6530e4865dc31bf5 not found: ID does not exist"
Sep 11 18:50:17.469953 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.469913    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dd795d9-bqnzv"]
Sep 11 18:50:17.472148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:17.472113    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6dd795d9-bqnzv"]
Sep 11 18:50:18.016049 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:18.016010    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:50:19.177650 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.177623    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:50:19.183780 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.183753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-credential-socket\") pod \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") "
Sep 11 18:50:19.183906 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.183795    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istiod-ca-cert\") pod \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") "
Sep 11 18:50:19.183906 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.183818    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-envoy\") pod \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") "
Sep 11 18:50:19.184030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.183944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6wqv\" (UniqueName: \"kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-kube-api-access-v6wqv\") pod \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") "
Sep 11 18:50:19.184030 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184002    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-token\") pod \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") "
Sep 11 18:50:19.184136 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e33e9b58-4ac7-4053-915a-d5f2d8172edc" (UID: "e33e9b58-4ac7-4053-915a-d5f2d8172edc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:19.184136 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184033    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-certs\") pod \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") "
Sep 11 18:50:19.184136 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-data\") pod \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") "
Sep 11 18:50:19.184136 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184097    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-socket\") pod \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") "
Sep 11 18:50:19.184136 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184134    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-podinfo\") pod \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\" (UID: \"e33e9b58-4ac7-4053-915a-d5f2d8172edc\") "
Sep 11 18:50:19.184405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e33e9b58-4ac7-4053-915a-d5f2d8172edc" (UID: "e33e9b58-4ac7-4053-915a-d5f2d8172edc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:50:19.184405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184283    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e33e9b58-4ac7-4053-915a-d5f2d8172edc" (UID: "e33e9b58-4ac7-4053-915a-d5f2d8172edc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:19.184520 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184479    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-credential-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:19.184520 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184498    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istiod-ca-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:19.184520 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e33e9b58-4ac7-4053-915a-d5f2d8172edc" (UID: "e33e9b58-4ac7-4053-915a-d5f2d8172edc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:19.184698 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184512    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:19.184698 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.184585    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-data" (OuterVolumeSpecName: "istio-data") pod "e33e9b58-4ac7-4053-915a-d5f2d8172edc" (UID: "e33e9b58-4ac7-4053-915a-d5f2d8172edc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:19.186100 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.186076    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e33e9b58-4ac7-4053-915a-d5f2d8172edc" (UID: "e33e9b58-4ac7-4053-915a-d5f2d8172edc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:19.186244 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.186189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-token" (OuterVolumeSpecName: "istio-token") pod "e33e9b58-4ac7-4053-915a-d5f2d8172edc" (UID: "e33e9b58-4ac7-4053-915a-d5f2d8172edc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:19.186244 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.186218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-kube-api-access-v6wqv" (OuterVolumeSpecName: "kube-api-access-v6wqv") pod "e33e9b58-4ac7-4053-915a-d5f2d8172edc" (UID: "e33e9b58-4ac7-4053-915a-d5f2d8172edc"). InnerVolumeSpecName "kube-api-access-v6wqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:19.186244 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.186231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e33e9b58-4ac7-4053-915a-d5f2d8172edc" (UID: "e33e9b58-4ac7-4053-915a-d5f2d8172edc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:50:19.284976 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.284877    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-envoy\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:19.284976 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.284916    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6wqv\" (UniqueName: \"kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-kube-api-access-v6wqv\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:19.284976 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.284927    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-token\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:19.284976 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.284936    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-data\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:19.284976 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.284944    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33e9b58-4ac7-4053-915a-d5f2d8172edc-workload-socket\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:19.284976 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.284951    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33e9b58-4ac7-4053-915a-d5f2d8172edc-istio-podinfo\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:50:19.396521 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.396483    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f" path="/var/lib/kubelet/pods/5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f/volumes"
Sep 11 18:50:19.455245 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.455211    2580 generic.go:356] "Generic (PLEG): container finished" podID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerID="4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f" exitCode=0
Sep 11 18:50:19.455400 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.455261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" event={"ID":"e33e9b58-4ac7-4053-915a-d5f2d8172edc","Type":"ContainerDied","Data":"4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f"}
Sep 11 18:50:19.455400 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.455282    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"
Sep 11 18:50:19.455400 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.455315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw" event={"ID":"e33e9b58-4ac7-4053-915a-d5f2d8172edc","Type":"ContainerDied","Data":"593b79f20ad94fddc7b3e90559e9e7b1faa581aae8aa3374fca64a0707021ec7"}
Sep 11 18:50:19.455400 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.455345    2580 scope.go:117] "RemoveContainer" containerID="4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f"
Sep 11 18:50:19.464615 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.464528    2580 scope.go:117] "RemoveContainer" containerID="c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e"
Sep 11 18:50:19.465142 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:50:19.465109    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f\": container with ID starting with 4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f not found: ID does not exist" containerID="4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f"
Sep 11 18:50:19.465261 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.465153    2580 image_gc_manager.go:222] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f\": container with ID starting with 4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f not found: ID does not exist"
Sep 11 18:50:19.473784 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.473754    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"]
Sep 11 18:50:19.474037 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.474019    2580 scope.go:117] "RemoveContainer" containerID="4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f"
Sep 11 18:50:19.474313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.474280    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f"} err="failed to get container status \"4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f\": rpc error: code = NotFound desc = could not find container \"4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f\": container with ID starting with 4d5073448e74b925c0bdcfc4b5c2e02b0fc44a08c162cceda37f3b37b09fd32f not found: ID does not exist"
Sep 11 18:50:19.474313 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.474314    2580 scope.go:117] "RemoveContainer" containerID="c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e"
Sep 11 18:50:19.474588 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:50:19.474564    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e\": container with ID starting with c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e not found: ID does not exist" containerID="c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e"
Sep 11 18:50:19.474644 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.474596    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e"} err="failed to get container status \"c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e\": rpc error: code = NotFound desc = could not find container \"c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e\": container with ID starting with c0b03e432cf386b0a8d7ade9cedc3707829403bea0976ae24aef5ea3716f078e not found: ID does not exist"
Sep 11 18:50:19.477876 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:19.477853    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-711a34-openshift-default-7dc78769d4-zcmkw"]
Sep 11 18:50:21.394800 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:50:21.394765    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" path="/var/lib/kubelet/pods/e33e9b58-4ac7-4053-915a-d5f2d8172edc/volumes"
Sep 11 18:51:11.152476 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:11.152428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:51:18.277348 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277311    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dd795d9-8fxqf"]
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277611    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277622    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277637    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f" containerName="manager"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277643    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f" containerName="manager"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277654    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277659    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277668    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277673    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277681    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f636f1d-8dd3-472d-822f-cab7cc6c790c" containerName="maas-api"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277686    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f636f1d-8dd3-472d-822f-cab7cc6c790c" containerName="maas-api"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277693    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277698    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277706    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277711    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277717    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d678cf4-0f5d-41c4-b19f-535da8abbf83" containerName="maas-api"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d678cf4-0f5d-41c4-b19f-535da8abbf83" containerName="maas-api"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277729    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277734    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277741    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy"
Sep 11 18:51:18.277737 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277746    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy"
Sep 11 18:51:18.278378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277793    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy"
Sep 11 18:51:18.278378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277800    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy"
Sep 11 18:51:18.278378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277806    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy"
Sep 11 18:51:18.278378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277813    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33e9b58-4ac7-4053-915a-d5f2d8172edc" containerName="istio-proxy"
Sep 11 18:51:18.278378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277818    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b4eedc1-dda7-4d8c-bc45-156bfcbc2b5f" containerName="manager"
Sep 11 18:51:18.278378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277823    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b05ed8b-fd62-461c-8d2e-bc3307dd7ffe" containerName="istio-proxy"
Sep 11 18:51:18.278378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277830    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy"
Sep 11 18:51:18.278378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277835    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f636f1d-8dd3-472d-822f-cab7cc6c790c" containerName="maas-api"
Sep 11 18:51:18.278378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277841    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy"
Sep 11 18:51:18.278378 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.277845    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d678cf4-0f5d-41c4-b19f-535da8abbf83" containerName="maas-api"
Sep 11 18:51:18.280704 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.280682    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.284326 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.284298    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 18:51:18.284326 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.284317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jg26l\""
Sep 11 18:51:18.284559 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.284404    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:51:18.284559 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.284551    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 18:51:18.289104 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.289052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd795d9-8fxqf"]
Sep 11 18:51:18.382178 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.382124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-webhook-cert\") pod \"maas-controller-6dd795d9-8fxqf\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") " pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.382178 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.382183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75j6\" (UniqueName: \"kubernetes.io/projected/d412b00b-76c7-4bd7-bc1a-0974db26d715-kube-api-access-g75j6\") pod \"maas-controller-6dd795d9-8fxqf\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") " pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.382423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.382281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-metrics-certs\") pod \"maas-controller-6dd795d9-8fxqf\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") " pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.483139 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.483101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-webhook-cert\") pod \"maas-controller-6dd795d9-8fxqf\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") " pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.483261 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.483149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75j6\" (UniqueName: \"kubernetes.io/projected/d412b00b-76c7-4bd7-bc1a-0974db26d715-kube-api-access-g75j6\") pod \"maas-controller-6dd795d9-8fxqf\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") " pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.483261 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.483173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-metrics-certs\") pod \"maas-controller-6dd795d9-8fxqf\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") " pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.485643 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.485607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-metrics-certs\") pod \"maas-controller-6dd795d9-8fxqf\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") " pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.485786 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.485661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-webhook-cert\") pod \"maas-controller-6dd795d9-8fxqf\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") " pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.493416 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.493389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75j6\" (UniqueName: \"kubernetes.io/projected/d412b00b-76c7-4bd7-bc1a-0974db26d715-kube-api-access-g75j6\") pod \"maas-controller-6dd795d9-8fxqf\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") " pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.592643 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.592550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:18.866002 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.865976    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd795d9-8fxqf"]
Sep 11 18:51:18.868645 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:51:18.868620    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd412b00b_76c7_4bd7_bc1a_0974db26d715.slice/crio-36015d0f8b3c1e6204d1577243d4a015a25579d47c12f62c92d4b0fcb187a6e8 WatchSource:0}: Error finding container 36015d0f8b3c1e6204d1577243d4a015a25579d47c12f62c92d4b0fcb187a6e8: Status 404 returned error can't find the container with id 36015d0f8b3c1e6204d1577243d4a015a25579d47c12f62c92d4b0fcb187a6e8
Sep 11 18:51:18.869825 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:18.869805    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:51:19.344666 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:19.344637    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:51:19.663776 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:19.663737    2580 scope.go:117] "RemoveContainer" containerID="539656bd71b26de6a3737e96ad8c5b46168b69e147e1f3cdc0fdc13654924d8f"
Sep 11 18:51:19.722051 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:19.722012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd795d9-8fxqf" event={"ID":"d412b00b-76c7-4bd7-bc1a-0974db26d715","Type":"ContainerStarted","Data":"5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d"}
Sep 11 18:51:19.722240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:19.722064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd795d9-8fxqf" event={"ID":"d412b00b-76c7-4bd7-bc1a-0974db26d715","Type":"ContainerStarted","Data":"36015d0f8b3c1e6204d1577243d4a015a25579d47c12f62c92d4b0fcb187a6e8"}
Sep 11 18:51:19.722240 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:19.722218    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:19.740496 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:19.740437    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6dd795d9-8fxqf" podStartSLOduration=1.268687605 podStartE2EDuration="1.740417598s" totalImagesPullingTime="471.729993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:51:19.739077832 +0000 UTC m=+2160.835671462" watchObservedRunningTime="2026-09-11 18:51:19.740417598 +0000 UTC m=+2160.837011242"
Sep 11 18:51:30.728221 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:30.728165    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:39.963082 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:39.963042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:51:40.056442 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.056405    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dd795d9-8fxqf"]
Sep 11 18:51:40.056638 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.056614    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6dd795d9-8fxqf" podUID="d412b00b-76c7-4bd7-bc1a-0974db26d715" containerName="manager" containerID="cri-o://5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d" gracePeriod=30
Sep 11 18:51:40.305230 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.305183    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:40.366004 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.365967    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-webhook-cert\") pod \"d412b00b-76c7-4bd7-bc1a-0974db26d715\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") "
Sep 11 18:51:40.366186 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.366014    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g75j6\" (UniqueName: \"kubernetes.io/projected/d412b00b-76c7-4bd7-bc1a-0974db26d715-kube-api-access-g75j6\") pod \"d412b00b-76c7-4bd7-bc1a-0974db26d715\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") "
Sep 11 18:51:40.366186 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.366053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-metrics-certs\") pod \"d412b00b-76c7-4bd7-bc1a-0974db26d715\" (UID: \"d412b00b-76c7-4bd7-bc1a-0974db26d715\") "
Sep 11 18:51:40.368364 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.368329    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d412b00b-76c7-4bd7-bc1a-0974db26d715-kube-api-access-g75j6" (OuterVolumeSpecName: "kube-api-access-g75j6") pod "d412b00b-76c7-4bd7-bc1a-0974db26d715" (UID: "d412b00b-76c7-4bd7-bc1a-0974db26d715"). InnerVolumeSpecName "kube-api-access-g75j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:51:40.368530 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.368433    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d412b00b-76c7-4bd7-bc1a-0974db26d715" (UID: "d412b00b-76c7-4bd7-bc1a-0974db26d715"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:51:40.368530 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.368496    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d412b00b-76c7-4bd7-bc1a-0974db26d715" (UID: "d412b00b-76c7-4bd7-bc1a-0974db26d715"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:51:40.467243 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.467173    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-webhook-cert\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:51:40.467243 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.467234    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75j6\" (UniqueName: \"kubernetes.io/projected/d412b00b-76c7-4bd7-bc1a-0974db26d715-kube-api-access-g75j6\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:51:40.467243 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.467245    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d412b00b-76c7-4bd7-bc1a-0974db26d715-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:51:40.819228 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.819170    2580 generic.go:356] "Generic (PLEG): container finished" podID="d412b00b-76c7-4bd7-bc1a-0974db26d715" containerID="5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d" exitCode=0
Sep 11 18:51:40.819442 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.819246    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd795d9-8fxqf"
Sep 11 18:51:40.819442 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.819236    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd795d9-8fxqf" event={"ID":"d412b00b-76c7-4bd7-bc1a-0974db26d715","Type":"ContainerDied","Data":"5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d"}
Sep 11 18:51:40.819442 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.819373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd795d9-8fxqf" event={"ID":"d412b00b-76c7-4bd7-bc1a-0974db26d715","Type":"ContainerDied","Data":"36015d0f8b3c1e6204d1577243d4a015a25579d47c12f62c92d4b0fcb187a6e8"}
Sep 11 18:51:40.819442 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.819396    2580 scope.go:117] "RemoveContainer" containerID="5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d"
Sep 11 18:51:40.828483 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.828455    2580 scope.go:117] "RemoveContainer" containerID="5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d"
Sep 11 18:51:40.828821 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:51:40.828801    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d\": container with ID starting with 5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d not found: ID does not exist" containerID="5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d"
Sep 11 18:51:40.828868 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.828832    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d"} err="failed to get container status \"5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d\": rpc error: code = NotFound desc = could not find container \"5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d\": container with ID starting with 5652b9052ff3b965f13d4af8bd1a647881398968080e1dcdb35bd93ede896a1d not found: ID does not exist"
Sep 11 18:51:40.840755 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.840720    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dd795d9-8fxqf"]
Sep 11 18:51:40.844906 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:40.844874    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6dd795d9-8fxqf"]
Sep 11 18:51:41.400334 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.400296    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d412b00b-76c7-4bd7-bc1a-0974db26d715" path="/var/lib/kubelet/pods/d412b00b-76c7-4bd7-bc1a-0974db26d715/volumes"
Sep 11 18:51:41.491287 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.491249    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:51:41.601128 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.601086    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dd795d9-6z4sg"]
Sep 11 18:51:41.601453 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.601439    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy"
Sep 11 18:51:41.601504 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.601455    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbda8b7d-4c3b-498f-bc1a-5995a49c41db" containerName="istio-proxy"
Sep 11 18:51:41.601504 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.601471    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d412b00b-76c7-4bd7-bc1a-0974db26d715" containerName="manager"
Sep 11 18:51:41.601504 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.601477    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d412b00b-76c7-4bd7-bc1a-0974db26d715" containerName="manager"
Sep 11 18:51:41.601599 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.601534    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d412b00b-76c7-4bd7-bc1a-0974db26d715" containerName="manager"
Sep 11 18:51:41.601599 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.601545    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8148e2-33d9-454e-a4e9-4dc11caa88c9" containerName="istio-proxy"
Sep 11 18:51:41.606027 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.605998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:41.608725 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.608689    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:51:41.608905 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.608748    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jg26l\""
Sep 11 18:51:41.608905 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.608794    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 18:51:41.608905 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.608748    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 18:51:41.613405 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.613379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd795d9-6z4sg"]
Sep 11 18:51:41.677191 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.677087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wvbq\" (UniqueName: \"kubernetes.io/projected/629de526-7704-49ea-a0e6-89596d7cf1ab-kube-api-access-2wvbq\") pod \"maas-controller-6dd795d9-6z4sg\" (UID: \"629de526-7704-49ea-a0e6-89596d7cf1ab\") " pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:41.677191 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.677154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/629de526-7704-49ea-a0e6-89596d7cf1ab-webhook-cert\") pod \"maas-controller-6dd795d9-6z4sg\" (UID: \"629de526-7704-49ea-a0e6-89596d7cf1ab\") " pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:41.677420 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.677229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629de526-7704-49ea-a0e6-89596d7cf1ab-metrics-certs\") pod \"maas-controller-6dd795d9-6z4sg\" (UID: \"629de526-7704-49ea-a0e6-89596d7cf1ab\") " pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:41.778333 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.778296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wvbq\" (UniqueName: \"kubernetes.io/projected/629de526-7704-49ea-a0e6-89596d7cf1ab-kube-api-access-2wvbq\") pod \"maas-controller-6dd795d9-6z4sg\" (UID: \"629de526-7704-49ea-a0e6-89596d7cf1ab\") " pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:41.778496 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.778372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/629de526-7704-49ea-a0e6-89596d7cf1ab-webhook-cert\") pod \"maas-controller-6dd795d9-6z4sg\" (UID: \"629de526-7704-49ea-a0e6-89596d7cf1ab\") " pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:41.778496 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.778402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629de526-7704-49ea-a0e6-89596d7cf1ab-metrics-certs\") pod \"maas-controller-6dd795d9-6z4sg\" (UID: \"629de526-7704-49ea-a0e6-89596d7cf1ab\") " pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:41.781071 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.781034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629de526-7704-49ea-a0e6-89596d7cf1ab-metrics-certs\") pod \"maas-controller-6dd795d9-6z4sg\" (UID: \"629de526-7704-49ea-a0e6-89596d7cf1ab\") " pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:41.781071 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.781064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/629de526-7704-49ea-a0e6-89596d7cf1ab-webhook-cert\") pod \"maas-controller-6dd795d9-6z4sg\" (UID: \"629de526-7704-49ea-a0e6-89596d7cf1ab\") " pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:41.787063 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.787034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wvbq\" (UniqueName: \"kubernetes.io/projected/629de526-7704-49ea-a0e6-89596d7cf1ab-kube-api-access-2wvbq\") pod \"maas-controller-6dd795d9-6z4sg\" (UID: \"629de526-7704-49ea-a0e6-89596d7cf1ab\") " pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:41.917330 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:41.917290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:42.188451 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:42.188219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd795d9-6z4sg"]
Sep 11 18:51:42.190958 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:51:42.190929    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629de526_7704_49ea_a0e6_89596d7cf1ab.slice/crio-574defb6bdcc5c15ecd65aa5e0be76f458d0f871e3b904bb5bea84e1b4cf2d34 WatchSource:0}: Error finding container 574defb6bdcc5c15ecd65aa5e0be76f458d0f871e3b904bb5bea84e1b4cf2d34: Status 404 returned error can't find the container with id 574defb6bdcc5c15ecd65aa5e0be76f458d0f871e3b904bb5bea84e1b4cf2d34
Sep 11 18:51:42.831723 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:42.831688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd795d9-6z4sg" event={"ID":"629de526-7704-49ea-a0e6-89596d7cf1ab","Type":"ContainerStarted","Data":"1f74aa3bd5bbbffad4fa9e69250e007690262e954c7fa82bf01c9208b700c1e5"}
Sep 11 18:51:42.832166 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:42.831739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd795d9-6z4sg" event={"ID":"629de526-7704-49ea-a0e6-89596d7cf1ab","Type":"ContainerStarted","Data":"574defb6bdcc5c15ecd65aa5e0be76f458d0f871e3b904bb5bea84e1b4cf2d34"}
Sep 11 18:51:42.832166 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:42.831831    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:51:42.848844 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:42.848785    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6dd795d9-6z4sg" podStartSLOduration=1.375957605 podStartE2EDuration="1.848766472s" totalImagesPullingTime="472.808867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:51:42.848613471 +0000 UTC m=+2183.945207103" watchObservedRunningTime="2026-09-11 18:51:42.848766472 +0000 UTC m=+2183.945360110"
Sep 11 18:51:53.839691 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:51:53.839657    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dd795d9-6z4sg"
Sep 11 18:52:19.682949 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:52:19.682911    2580 scope.go:117] "RemoveContainer" containerID="d56fcb7526220fb5c67e048d78a57926366ea3a996aa67202e9ee8fda7f8f26b"
Sep 11 18:53:14.466400 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:53:14.466363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:53:45.475712 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:53:45.475672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:53:45.578474 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:53:45.578437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:54:14.200612 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:54:14.200530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:54:14.303220 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:54:14.303163    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:54:26.367417 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:54:26.367382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:54:31.458377 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:54:31.458337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:54:31.558794 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:54:31.558761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:54:36.976607 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:54:36.976568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:54:49.292647 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:54:49.292605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:54:49.354801 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:54:49.354773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:54:53.380716 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:54:53.380677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:54:54.777144 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:54:54.777106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:55:12.815552 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.815506    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"]
Sep 11 18:55:12.819239 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.819194    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.829059 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.829018    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"]
Sep 11 18:55:12.863486 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.863441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db859283-6993-44ba-b32e-4a43b50a8fde-openshift-service-ca\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.863671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.863505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db859283-6993-44ba-b32e-4a43b50a8fde-maas-api-tls\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.863671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.863573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db859283-6993-44ba-b32e-4a43b50a8fde-metrics-certs\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.863671 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.863633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvng\" (UniqueName: \"kubernetes.io/projected/db859283-6993-44ba-b32e-4a43b50a8fde-kube-api-access-9fvng\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.964878 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.964830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db859283-6993-44ba-b32e-4a43b50a8fde-openshift-service-ca\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.965082 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.964904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db859283-6993-44ba-b32e-4a43b50a8fde-maas-api-tls\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.965082 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.964937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db859283-6993-44ba-b32e-4a43b50a8fde-metrics-certs\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.965082 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.964988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvng\" (UniqueName: \"kubernetes.io/projected/db859283-6993-44ba-b32e-4a43b50a8fde-kube-api-access-9fvng\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.965741 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.965707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db859283-6993-44ba-b32e-4a43b50a8fde-openshift-service-ca\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.967560 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.967539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db859283-6993-44ba-b32e-4a43b50a8fde-metrics-certs\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.967685 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.967663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db859283-6993-44ba-b32e-4a43b50a8fde-maas-api-tls\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:12.973738 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:12.973711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvng\" (UniqueName: \"kubernetes.io/projected/db859283-6993-44ba-b32e-4a43b50a8fde-kube-api-access-9fvng\") pod \"maas-api-798f7d6bf7-zbjmd\" (UID: \"db859283-6993-44ba-b32e-4a43b50a8fde\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:13.131906 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:13.131863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:13.402718 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:13.402681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"]
Sep 11 18:55:13.405890 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:55:13.405860    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb859283_6993_44ba_b32e_4a43b50a8fde.slice/crio-95f70c11357e0ee89d40ff7923b71723222b93277802f320efffe700cda6a8ee WatchSource:0}: Error finding container 95f70c11357e0ee89d40ff7923b71723222b93277802f320efffe700cda6a8ee: Status 404 returned error can't find the container with id 95f70c11357e0ee89d40ff7923b71723222b93277802f320efffe700cda6a8ee
Sep 11 18:55:13.749980 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:13.749881    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd" event={"ID":"db859283-6993-44ba-b32e-4a43b50a8fde","Type":"ContainerStarted","Data":"95f70c11357e0ee89d40ff7923b71723222b93277802f320efffe700cda6a8ee"}
Sep 11 18:55:14.756879 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:14.756838    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd" event={"ID":"db859283-6993-44ba-b32e-4a43b50a8fde","Type":"ContainerStarted","Data":"d0ce6420fc650179f6e4a5c173375e96f7c052524650b7366e3fee5375adb2d4"}
Sep 11 18:55:14.757320 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:14.756930    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:14.774403 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:14.774342    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd" podStartSLOduration=2.092143597 podStartE2EDuration="2.774323684s" totalImagesPullingTime="682.180087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:55:14.772721055 +0000 UTC m=+2395.869314690" watchObservedRunningTime="2026-09-11 18:55:14.774323684 +0000 UTC m=+2395.870917315"
Sep 11 18:55:20.767281 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:20.767249    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-zbjmd"
Sep 11 18:55:20.816043 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:20.816001    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"]
Sep 11 18:55:20.816276 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:20.816232    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc" podUID="b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae" containerName="maas-api" containerID="cri-o://d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b" gracePeriod=30
Sep 11 18:55:22.055553 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.055522    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:55:22.141803 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.141770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-maas-api-tls\") pod \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") "
Sep 11 18:55:22.142003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.141821    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bj6k\" (UniqueName: \"kubernetes.io/projected/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-kube-api-access-8bj6k\") pod \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") "
Sep 11 18:55:22.142003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.141861    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-openshift-service-ca\") pod \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") "
Sep 11 18:55:22.142003 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.141901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-metrics-certs\") pod \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\" (UID: \"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae\") "
Sep 11 18:55:22.142312 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.142282    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae" (UID: "b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:55:22.144025 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.143990    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae" (UID: "b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:55:22.144025 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.144004    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae" (UID: "b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:55:22.144157 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.144043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-kube-api-access-8bj6k" (OuterVolumeSpecName: "kube-api-access-8bj6k") pod "b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae" (UID: "b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae"). InnerVolumeSpecName "kube-api-access-8bj6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:55:22.242539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.242446    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bj6k\" (UniqueName: \"kubernetes.io/projected/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-kube-api-access-8bj6k\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:55:22.242539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.242476    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-openshift-service-ca\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:55:22.242539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.242488    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-metrics-certs\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:55:22.242539 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.242496    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae-maas-api-tls\") on node \"ip-10-0-129-7.ec2.internal\" DevicePath \"\""
Sep 11 18:55:22.792751 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.792713    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae" containerID="d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b" exitCode=0
Sep 11 18:55:22.792936 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.792761    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"
Sep 11 18:55:22.792936 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.792801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc" event={"ID":"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae","Type":"ContainerDied","Data":"d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b"}
Sep 11 18:55:22.792936 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.792857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc" event={"ID":"b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae","Type":"ContainerDied","Data":"87f0b6e207c0f0dde6552cdd73514211605d9d168601b2bd85f8be1d5ccabbed"}
Sep 11 18:55:22.792936 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.792884    2580 scope.go:117] "RemoveContainer" containerID="d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b"
Sep 11 18:55:22.808266 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.808243    2580 scope.go:117] "RemoveContainer" containerID="d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b"
Sep 11 18:55:22.808603 ip-10-0-129-7 kubenswrapper[2580]: E0911 18:55:22.808584    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b\": container with ID starting with d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b not found: ID does not exist" containerID="d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b"
Sep 11 18:55:22.808691 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.808614    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b"} err="failed to get container status \"d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b\": rpc error: code = NotFound desc = could not find container \"d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b\": container with ID starting with d497fa3cfd5ccdfd59d88f96291d9ad25564e88f1e48d2096927bc16cac8863b not found: ID does not exist"
Sep 11 18:55:22.825923 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.825884    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"]
Sep 11 18:55:22.837264 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:22.837227    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-5spcc"]
Sep 11 18:55:23.395058 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:23.395015    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae" path="/var/lib/kubelet/pods/b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae/volumes"
Sep 11 18:55:52.164916 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:52.164780    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dd795d9-6z4sg_629de526-7704-49ea-a0e6-89596d7cf1ab/manager/0.log"
Sep 11 18:55:52.631735 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:52.631706    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-ml5cm_362492d6-442f-48c8-a982-003efc62c937/copy-manifests/0.log"
Sep 11 18:55:52.645988 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:52.645951    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-ml5cm_362492d6-442f-48c8-a982-003efc62c937/manager/0.log"
Sep 11 18:55:53.223336 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:53.223304    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-798f7d6bf7-zbjmd_db859283-6993-44ba-b32e-4a43b50a8fde/maas-api/0.log"
Sep 11 18:55:53.550828 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:53.550733    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-92lgb_e866b959-13df-4f37-a273-749b4747d0b4/postgres/0.log"
Sep 11 18:55:55.473420 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:55.473392    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4mz5c_90d8bfb8-bff1-4ad4-8707-938e18139c93/discovery/0.log"
Sep 11 18:55:56.629478 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:56.629445    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-9hw2w_a439c393-bf3a-4de7-ad06-00ee2caf5b9f/main/0.log"
Sep 11 18:55:56.966804 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:55:56.966714    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx2ggg_5ae642ae-41a7-431f-8a78-b487d15d3fae/main/0.log"
Sep 11 18:56:02.281687 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:02.281649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:56:02.389631 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:02.389594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log" limit=1024
Sep 11 18:56:05.350533 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:05.350492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8s7n8_1649793f-bd76-49a7-9b4e-61473e8857b2/global-pull-secret-syncer/0.log"
Sep 11 18:56:05.492888 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:05.492856    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9zqsb_8fca03eb-a3d5-4625-b302-194c27bf4592/konnectivity-agent/0.log"
Sep 11 18:56:05.651844 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:05.651807    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-7.ec2.internal_d40969adc2806df4839ecaab1c9a6cc7/haproxy/0.log"
Sep 11 18:56:12.864436 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:12.864405    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2ltj_09f116e9-3034-4d86-9383-b25b73d9b254/node-exporter/0.log"
Sep 11 18:56:12.889741 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:12.889714    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2ltj_09f116e9-3034-4d86-9383-b25b73d9b254/kube-rbac-proxy/0.log"
Sep 11 18:56:12.916103 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:12.916074    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2ltj_09f116e9-3034-4d86-9383-b25b73d9b254/init-textfile/0.log"
Sep 11 18:56:13.613412 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:13.613379    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-857594858f-cvgsj_9932a593-5517-4a43-b4be-952219bb1435/thanos-query/0.log"
Sep 11 18:56:13.641073 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:13.641041    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-857594858f-cvgsj_9932a593-5517-4a43-b4be-952219bb1435/kube-rbac-proxy-web/0.log"
Sep 11 18:56:13.664358 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:13.664332    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-857594858f-cvgsj_9932a593-5517-4a43-b4be-952219bb1435/kube-rbac-proxy/0.log"
Sep 11 18:56:13.685798 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:13.685763    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-857594858f-cvgsj_9932a593-5517-4a43-b4be-952219bb1435/prom-label-proxy/0.log"
Sep 11 18:56:13.710452 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:13.710420    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-857594858f-cvgsj_9932a593-5517-4a43-b4be-952219bb1435/kube-rbac-proxy-rules/0.log"
Sep 11 18:56:13.736930 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:13.736898    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-857594858f-cvgsj_9932a593-5517-4a43-b4be-952219bb1435/kube-rbac-proxy-metrics/0.log"
Sep 11 18:56:15.056288 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.056254    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-grkvb_0a73a24f-fb75-42c8-96cb-8e8a809c586d/networking-console-plugin/0.log"
Sep 11 18:56:15.588858 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.588825    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"]
Sep 11 18:56:15.589142 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.589130    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae" containerName="maas-api"
Sep 11 18:56:15.589186 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.589143    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae" containerName="maas-api"
Sep 11 18:56:15.589271 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.589259    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d9a2f8-9fec-440e-8f95-9c5cf5dbc6ae" containerName="maas-api"
Sep 11 18:56:15.592500 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.592475    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.594780 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.594757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kfqdv\"/\"default-dockercfg-kn4ch\""
Sep 11 18:56:15.594780 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.594775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfqdv\"/\"kube-root-ca.crt\""
Sep 11 18:56:15.595472 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.595454    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfqdv\"/\"openshift-service-ca.crt\""
Sep 11 18:56:15.602286 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.602260    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"]
Sep 11 18:56:15.704229 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.704159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ffs\" (UniqueName: \"kubernetes.io/projected/6d83c033-7857-41e4-bc59-c080dc5319ad-kube-api-access-p4ffs\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.704423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.704277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-podres\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.704423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.704319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-proc\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.704423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.704374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-sys\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.704423 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.704408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-lib-modules\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.805107 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.805065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ffs\" (UniqueName: \"kubernetes.io/projected/6d83c033-7857-41e4-bc59-c080dc5319ad-kube-api-access-p4ffs\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.805309 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.805130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-podres\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.805309 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.805158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-proc\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.805309 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.805240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-sys\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.805309 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.805262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-lib-modules\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.805484 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.805342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-proc\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.805484 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.805355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-sys\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.805484 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.805343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-podres\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.805484 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.805414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d83c033-7857-41e4-bc59-c080dc5319ad-lib-modules\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.814011 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.813983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ffs\" (UniqueName: \"kubernetes.io/projected/6d83c033-7857-41e4-bc59-c080dc5319ad-kube-api-access-p4ffs\") pod \"perf-node-gather-daemonset-qkl5m\" (UID: \"6d83c033-7857-41e4-bc59-c080dc5319ad\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:15.902167 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:15.902129    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:16.170731 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:16.170651    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"]
Sep 11 18:56:16.174449 ip-10-0-129-7 kubenswrapper[2580]: W0911 18:56:16.174413    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6d83c033_7857_41e4_bc59_c080dc5319ad.slice/crio-7f0d77006d7cbd1208ed3222fa4c848a5644cb11cf770765157ff625a018404d WatchSource:0}: Error finding container 7f0d77006d7cbd1208ed3222fa4c848a5644cb11cf770765157ff625a018404d: Status 404 returned error can't find the container with id 7f0d77006d7cbd1208ed3222fa4c848a5644cb11cf770765157ff625a018404d
Sep 11 18:56:17.038382 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:17.038345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m" event={"ID":"6d83c033-7857-41e4-bc59-c080dc5319ad","Type":"ContainerStarted","Data":"ace61053da9fc35110db68af63640456ae3529e1566c76cc2925bb056d5918ca"}
Sep 11 18:56:17.038567 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:17.038397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m" event={"ID":"6d83c033-7857-41e4-bc59-c080dc5319ad","Type":"ContainerStarted","Data":"7f0d77006d7cbd1208ed3222fa4c848a5644cb11cf770765157ff625a018404d"}
Sep 11 18:56:17.038567 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:17.038427    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:17.055109 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:17.055058    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m" podStartSLOduration=2.055031114 podStartE2EDuration="2.055031114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:56:17.053832202 +0000 UTC m=+2458.150425833" watchObservedRunningTime="2026-09-11 18:56:17.055031114 +0000 UTC m=+2458.151624781"
Sep 11 18:56:18.343148 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:18.343115    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h87pc_030d0519-adf2-40e3-a456-7d80868f9725/dns/0.log"
Sep 11 18:56:18.365372 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:18.365344    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h87pc_030d0519-adf2-40e3-a456-7d80868f9725/kube-rbac-proxy/0.log"
Sep 11 18:56:18.467692 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:18.467666    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cqq8n_355ecf34-6283-43ff-9bbc-1313be1dea3a/dns-node-resolver/0.log"
Sep 11 18:56:19.107949 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:19.107920    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrdkw_adbddaee-40dd-465e-bd6c-7a32d384346a/node-ca/0.log"
Sep 11 18:56:20.553780 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:20.553743    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4mz5c_90d8bfb8-bff1-4ad4-8707-938e18139c93/discovery/0.log"
Sep 11 18:56:21.688631 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:21.688598    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rnzmv_15a582f5-b3ab-4ba3-aed1-6f86d5308a2f/serve-healthcheck-canary/0.log"
Sep 11 18:56:22.388706 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:22.388675    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nr8s4_5c53d821-2989-42fb-8d84-07d413795f9c/kube-rbac-proxy/0.log"
Sep 11 18:56:22.412922 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:22.412895    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nr8s4_5c53d821-2989-42fb-8d84-07d413795f9c/exporter/0.log"
Sep 11 18:56:22.438432 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:22.438402    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nr8s4_5c53d821-2989-42fb-8d84-07d413795f9c/extractor/0.log"
Sep 11 18:56:23.052738 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:23.052708    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-qkl5m"
Sep 11 18:56:25.492228 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:25.492177    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dd795d9-6z4sg_629de526-7704-49ea-a0e6-89596d7cf1ab/manager/0.log"
Sep 11 18:56:25.844532 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:25.844443    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-ml5cm_362492d6-442f-48c8-a982-003efc62c937/manager/0.log"
Sep 11 18:56:25.861869 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:25.861837    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-ml5cm_362492d6-442f-48c8-a982-003efc62c937/copy-manifests/0.log"
Sep 11 18:56:27.088902 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:27.088864    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4mz5c_90d8bfb8-bff1-4ad4-8707-938e18139c93/discovery/0.log"
Sep 11 18:56:27.185685 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:27.185596    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5588b8499b-v8pc7_6f2df4b0-974f-43c7-a6b9-1fa3bb9c1a15/manager/0.log"
Sep 11 18:56:29.202141 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:29.202104    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4mz5c_90d8bfb8-bff1-4ad4-8707-938e18139c93/discovery/0.log"
Sep 11 18:56:32.384506 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:32.384473    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-xd46r_0d74c37f-f474-4ad1-b428-f4ebdea92260/migrator/0.log"
Sep 11 18:56:32.407675 ip-10-0-129-7 kubenswrapper[2580]: I0911 18:56:32.407643    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-xd46r_0d74c37f-f474-4ad1-b428-f4ebdea92260/graceful-termination/0.log"