Jul 28 20:08:20.272490 ip-10-0-140-243 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 20:08:20.644720 ip-10-0-140-243 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:08:20.644720 ip-10-0-140-243 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 20:08:20.644720 ip-10-0-140-243 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:08:20.644720 ip-10-0-140-243 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 20:08:20.644720 ip-10-0-140-243 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:08:20.646336 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.646253    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 20:08:20.650739 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650725    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:08:20.650739 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650739    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650744    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650747    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650750    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650755    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650758    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650761    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650763    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650767    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650770    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650773    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650776    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650779    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650782    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650785    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650788    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650791    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650794    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650802    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650805    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:08:20.650808 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650807    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650810    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650813    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650816    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650819    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650822    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650825    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650827    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650830    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650833    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650836    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650839    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650842    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650844    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650847    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650849    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650853    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650855    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650858    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650861    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:08:20.651296 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650863    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650866    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650868    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650871    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650873    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650876    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650878    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650882    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650886    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650889    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650891    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650894    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650896    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650899    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650903    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650907    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650910    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650913    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650916    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:08:20.651764 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650919    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650923    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650926    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650929    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650931    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650934    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650936    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650939    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650941    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650944    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650947    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650949    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650953    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650956    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650958    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650961    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650963    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650966    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650969    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650971    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:08:20.652242 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650974    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650976    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650979    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650982    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650984    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650987    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650991    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650995    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.650998    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651001    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651004    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651007    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651010    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651012    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651015    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651017    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651019    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651022    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651024    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:08:20.652732 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651745    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651751    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651754    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651757    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651760    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651762    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651767    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651770    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651773    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651775    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651778    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651781    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651784    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651786    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651789    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651792    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651795    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651797    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651800    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651802    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:08:20.653198 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651805    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651808    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651810    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651813    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651815    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651818    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651821    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651825    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651827    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651830    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651833    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651835    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651838    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651841    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651843    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651846    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651849    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651851    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651854    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651857    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:08:20.653708 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651859    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651862    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651864    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651867    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651871    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651874    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651876    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651879    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651882    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651885    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651887    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651890    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651893    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651895    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651898    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651900    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651903    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651905    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651908    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651910    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:08:20.654204 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651913    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651915    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651918    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651920    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651923    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651926    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651928    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651930    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651933    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651938    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651941    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651944    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651947    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651950    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651953    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651957    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651960    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651962    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651965    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651968    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:08:20.654697 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651970    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651973    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651976    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651980    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651983    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651985    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651988    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651990    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651993    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651995    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.651998    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652000    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652003    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652005    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652009    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652012    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652016    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652019    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652022    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:08:20.655180 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652397    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652402    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652405    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652408    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652411    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652414    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652417    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652422    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652425    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652428    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652432    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652436    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652439    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652442    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652445    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652448    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652451    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652453    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652457    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:08:20.655654 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652461    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652464    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652466    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652469    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652472    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652475    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652477    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652480    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652482    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652485    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652487    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652490    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652492    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652495    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652497    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652500    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652503    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652505    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652508    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:08:20.656143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652511    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652515    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652518    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652521    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652523    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652526    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652528    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652531    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652533    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652536    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652538    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652541    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652544    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652546    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652549    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652551    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652554    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652556    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652559    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652561    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:08:20.656632 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652563    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652566    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652568    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652571    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652574    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652577    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652580    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652582    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652585    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652587    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652589    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652592    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652594    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652599    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652601    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652604    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652606    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652609    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652611    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652614    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:08:20.657118 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652618    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652620    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652623    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652625    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652628    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652630    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652633    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652635    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652638    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652640    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652643    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652645    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652648    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652650    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652653    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652655    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652658    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652660    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652663    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652665    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:08:20.657661 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.652668    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652734    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652740    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652746    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652751    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652757    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652761    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652765    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652774    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652777    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652780    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652784    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652787    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652790    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652793    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652796    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652799    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652801    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652804    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652808    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652811    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652814    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652817    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652821    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 20:08:20.658145 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652825    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652827    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652830    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652834    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652837    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652839    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652842    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652845    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652848    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652852    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652855    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652858    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652861    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652865    2575 flags.go:64] FLAG: --enable-server="true"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652868    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652922    2575 flags.go:64] FLAG: --event-burst="100"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652927    2575 flags.go:64] FLAG: --event-qps="50"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652931    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652934    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652937    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652941    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652943    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652946    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652949    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652952    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 20:08:20.658778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652955    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652958    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652961    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652963    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652966    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652969    2575 flags.go:64] FLAG: --feature-gates=""
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652973    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652977    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652980    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652983    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652986    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652989    2575 flags.go:64] FLAG: --help="false"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652991    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-140-243.ec2.internal"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652994    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.652997    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653000    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653003    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653007    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653010    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653012    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653017    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653020    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653023    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653026    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 20:08:20.659403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653029    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653032    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653035    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653038    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653041    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653043    2575 flags.go:64] FLAG: --lock-file=""
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653046    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653049    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653052    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653058    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653060    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653063    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653066    2575 flags.go:64] FLAG: --logging-format="text"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653069    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653072    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653075    2575 flags.go:64] FLAG: --manifest-url=""
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653078    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653082    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653085    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653089    2575 flags.go:64] FLAG: --max-pods="110"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653092    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653094    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653097    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653100    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653103    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 20:08:20.660163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653106    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653109    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653116    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653120    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653123    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653126    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653129    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653135    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653137    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653140    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653143    2575 flags.go:64] FLAG: --port="10250"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653146    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653149    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0213f033e70189aac"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653152    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653155    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653157    2575 flags.go:64] FLAG: --register-node="true"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653160    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653164    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653167    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653169    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653172    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653175    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653178    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653181    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653184    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 20:08:20.661188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653187    2575 flags.go:64] FLAG: --runonce="false"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653190    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653193    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653196    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653199    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653202    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653205    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653208    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653211    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653234    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653238    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653241    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653244    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653247    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653250    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653254    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653259    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653262    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653265    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653270    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653272    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653275    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653278    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653281    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653284    2575 flags.go:64] FLAG: --v="2"
Jul 28 20:08:20.661839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653287    2575 flags.go:64] FLAG: --version="false"
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653291    2575 flags.go:64] FLAG: --vmodule=""
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653295    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653298    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653410    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653415    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653418    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653423    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653427    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653430    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653433    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653435    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653438    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653440    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653443    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653446    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653448    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653451    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653454    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653456    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653459    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:08:20.662539 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653462    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653464    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653467    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653470    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653473    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653475    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653479    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653483    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653486    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653489    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653492    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653494    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653497    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653499    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653502    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653505    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653507    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653510    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653514    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:08:20.663080 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653517    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653519    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653522    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653524    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653527    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653529    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653532    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653535    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653537    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653540    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653543    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653547    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653550    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653553    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653555    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653565    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653568    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653571    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653574    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653576    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:08:20.663568 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653579    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653581    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653584    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653586    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653589    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653591    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653594    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653596    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653599    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653601    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653604    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653607    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653610    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653612    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653615    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653618    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653622    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653625    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653628    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:08:20.664068 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653630    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653633    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653636    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653639    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653645    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653648    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653651    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653653    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653656    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653659    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653662    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653664    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653667    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653669    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653672    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653674    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653677    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653680    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653682    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:08:20.664550 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653684    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653687    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653689    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653692    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.653694    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.653701    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.661937    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.661951    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.661969    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.661975    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662029    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662034    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662037    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662040    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662043    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:08:20.665004 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662046    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662050    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662052    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662055    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662058    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662061    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662065    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662069    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662072    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662076    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662078    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662081    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662084    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662087    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662089    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662092    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662094    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662097    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662100    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:08:20.665412 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662103    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662107    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662110    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662112    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662116    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662119    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662122    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662125    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662127    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662130    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662133    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662136    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662138    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662142    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662146    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662149    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662152    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662154    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662157    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:08:20.665879 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662159    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662162    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662164    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662167    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662169    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662172    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662175    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662177    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662180    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662182    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662185    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662187    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662190    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662192    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662195    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662197    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662200    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662203    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662206    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662209    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:08:20.666440 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662212    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662214    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662217    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662220    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662238    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662243    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662247    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662250    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662253    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662256    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662258    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662261    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662264    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662267    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662269    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662272    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662276    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662278    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662281    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662284    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:08:20.667039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662286    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662289    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662291    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662294    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662296    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662299    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662301    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662304    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662306    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662309    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662312    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662314    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662317    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662320    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662323    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:08:20.667531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662325    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.662331    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662446    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662452    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662455    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662457    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662460    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662463    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662465    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662468    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662471    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662473    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662476    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662479    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662482    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:08:20.668074 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662485    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662487    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662490    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662492    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662495    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662497    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662500    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662503    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662505    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662508    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662510    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662513    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662516    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662519    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662521    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662524    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662526    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662529    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662531    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662534    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:08:20.668707 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662536    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662539    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662542    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662545    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662547    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662550    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662553    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662556    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662558    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662561    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662563    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662567    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662570    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662573    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662575    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662578    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662580    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662583    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662585    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662588    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:08:20.669375 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662590    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662593    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662596    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662599    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662601    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662604    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662606    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662609    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662611    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662614    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662616    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662619    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662621    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662625    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662629    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662631    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662634    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662637    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662641    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662644    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:08:20.669864 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662647    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662650    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662653    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662656    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662660    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662664    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662667    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662669    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662672    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662674    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662677    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662679    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662682    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662685    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662687    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662690    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662692    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662695    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662698    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662700    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:08:20.670397 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662703    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662706    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662708    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662710    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662713    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:20.662715    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.662720    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.662816    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.665187    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.666707    2575 server.go:1019] "Starting client certificate rotation"
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.666821    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.666863    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.669211    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 20:08:20.670918 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.670321    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 20:08:20.682867 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.682852    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 28 20:08:20.685263 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.685247    2575 log.go:25] "Validated CRI v1 image API"
Jul 28 20:08:20.686106 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.686091    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 20:08:20.690939 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.690917    2575 fs.go:135] Filesystem UUIDs: map[58a7d299-8613-4b9c-ab95-f818ff3719db:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a466adcb-5ace-44bd-bb61-41a1146a45d1:/dev/nvme0n1p4]
Jul 28 20:08:20.691010 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.690938    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 20:08:20.697322 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.697161    2575 manager.go:216] Machine: {Timestamp:2026-07-28 20:08:20.695914258 +0000 UTC m=+0.310151382 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101941 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c14bac8201e25225ae19fa06c0ec7 SystemUUID:ec2c14ba-c820-1e25-225a-e19fa06c0ec7 BootID:91b09cdb-0184-4832-8e6b-db161257969b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fd:c7:dc:cb:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fd:c7:dc:cb:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:fa:13:80:0b:8f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 20:08:20.697322 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.697318    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 20:08:20.697440 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.697415    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 20:08:20.699956 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.699938    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 20:08:20.700333 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.700312    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 20:08:20.700459 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.700335    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-243.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 20:08:20.701101 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.701092    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 20:08:20.701133 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.701105    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 20:08:20.701133 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.701112    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 20:08:20.701852 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.701839    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 20:08:20.702994 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.702984    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 20:08:20.704926 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.704915    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 20:08:20.705040 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.705023    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 20:08:20.705439 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.705430    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 20:08:20.705471 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.705457    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 20:08:20.705498 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.705477    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 20:08:20.705498 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.705487    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 28 20:08:20.705498 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.705496    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 20:08:20.706777 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.706760    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 20:08:20.706840 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.706790    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 20:08:20.711831 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.711811    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 20:08:20.714071 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.714057    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 20:08:20.714144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.714086    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 20:08:20.716170 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716158    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 20:08:20.716236 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716176    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 20:08:20.716236 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716184    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 20:08:20.716236 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716192    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 20:08:20.716236 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716203    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 20:08:20.716236 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716209    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 20:08:20.716236 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716216    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 20:08:20.716236 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716222    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 20:08:20.716460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716246    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 20:08:20.716460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716251    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 20:08:20.716460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716260    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 20:08:20.716460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.716265    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 20:08:20.717017 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.717004    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 20:08:20.717017 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.717015    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 20:08:20.717209 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.717173    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 20:08:20.717330 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.717210    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-243.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 20:08:20.719896 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.719885    2575 server.go:1268] "Started kubelet"
Jul 28 20:08:20.719979 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.719956    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 20:08:20.720037 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.719969    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 20:08:20.720083 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.720061    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 20:08:20.720684 ip-10-0-140-243 systemd[1]: Started Kubernetes Kubelet.
Jul 28 20:08:20.721954 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.721938    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 20:08:20.722293 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.722281    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 20:08:20.725405 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.725374    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 20:08:20.726767 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.726746    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-243.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 20:08:20.727685 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.726775    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-243.ec2.internal.18c68cf861d13449  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-243.ec2.internal,UID:ip-10-0-140-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-243.ec2.internal,},FirstTimestamp:2026-07-28 20:08:20.719858761 +0000 UTC m=+0.334095886,LastTimestamp:2026-07-28 20:08:20.719858761 +0000 UTC m=+0.334095886,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-243.ec2.internal,}"
Jul 28 20:08:20.730720 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.730706    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 20:08:20.730800 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.730722    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 20:08:20.730854 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.730831    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:20.731491 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731371    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 20:08:20.731491 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731388    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 20:08:20.731491 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731402    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 20:08:20.731491 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731490    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 20:08:20.731695 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731500    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 20:08:20.731695 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731559    2575 factory.go:55] Registering systemd factory
Jul 28 20:08:20.731695 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731571    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 28 20:08:20.731835 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731810    2575 factory.go:153] Registering CRI-O factory
Jul 28 20:08:20.731835 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731823    2575 factory.go:223] Registration of the crio container factory successfully
Jul 28 20:08:20.731931 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731878    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 20:08:20.731931 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731897    2575 factory.go:103] Registering Raw factory
Jul 28 20:08:20.731931 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.731907    2575 manager.go:1196] Started watching for new ooms in manager
Jul 28 20:08:20.732358 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.732347    2575 manager.go:349] Starting recovery of all containers
Jul 28 20:08:20.732568 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.732547    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-243.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 20:08:20.732838 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.732819    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 20:08:20.744589 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.744484    2575 manager.go:354] Recovery completed
Jul 28 20:08:20.748484 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.748471    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:08:20.750520 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.750492    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:08:20.750572 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.750540    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:08:20.750572 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.750557    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:08:20.751093 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.751076    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 20:08:20.751157 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.751093    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 20:08:20.751157 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.751111    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 20:08:20.751234 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.751082    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:08:20.751864 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.751850    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:08:20.751903 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.751875    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:08:20.751903 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.751888    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:08:20.753062 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.753052    2575 policy_none.go:49] "None policy: Start"
Jul 28 20:08:20.753147 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.753066    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 20:08:20.753147 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.753076    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 20:08:20.754307 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.754295    2575 policy_none.go:47] "Start"
Jul 28 20:08:20.759300 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.759235    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-243.ec2.internal.18c68cf863a51471  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-243.ec2.internal,UID:ip-10-0-140-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-243.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-243.ec2.internal,},FirstTimestamp:2026-07-28 20:08:20.750521457 +0000 UTC m=+0.364758586,LastTimestamp:2026-07-28 20:08:20.750521457 +0000 UTC m=+0.364758586,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-243.ec2.internal,}"
Jul 28 20:08:20.761816 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.761751    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-243.ec2.internal.18c68cf863a57813  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-243.ec2.internal,UID:ip-10-0-140-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-243.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-243.ec2.internal,},FirstTimestamp:2026-07-28 20:08:20.750546963 +0000 UTC m=+0.364784092,LastTimestamp:2026-07-28 20:08:20.750546963 +0000 UTC m=+0.364784092,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-243.ec2.internal,}"
Jul 28 20:08:20.769293 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.768222    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-243.ec2.internal.18c68cf863a5b4bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-243.ec2.internal,UID:ip-10-0-140-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-243.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-243.ec2.internal,},FirstTimestamp:2026-07-28 20:08:20.750562491 +0000 UTC m=+0.364799614,LastTimestamp:2026-07-28 20:08:20.750562491 +0000 UTC m=+0.364799614,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-243.ec2.internal,}"
Jul 28 20:08:20.772553 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.772490    2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-243.ec2.internal.18c68cf863a51471\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-243.ec2.internal.18c68cf863a51471  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-243.ec2.internal,UID:ip-10-0-140-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-243.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-243.ec2.internal,},FirstTimestamp:2026-07-28 20:08:20.750521457 +0000 UTC m=+0.364758586,LastTimestamp:2026-07-28 20:08:20.751862456 +0000 UTC m=+0.366099579,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-243.ec2.internal,}"
Jul 28 20:08:20.776817 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.776650    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h5k6s"
Jul 28 20:08:20.778980 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.778924    2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-243.ec2.internal.18c68cf863a57813\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-243.ec2.internal.18c68cf863a57813  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-243.ec2.internal,UID:ip-10-0-140-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-243.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-243.ec2.internal,},FirstTimestamp:2026-07-28 20:08:20.750546963 +0000 UTC m=+0.364784092,LastTimestamp:2026-07-28 20:08:20.751880905 +0000 UTC m=+0.366118030,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-243.ec2.internal,}"
Jul 28 20:08:20.781717 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.781639    2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-243.ec2.internal.18c68cf863a5b4bb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-243.ec2.internal.18c68cf863a5b4bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-243.ec2.internal,UID:ip-10-0-140-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-243.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-243.ec2.internal,},FirstTimestamp:2026-07-28 20:08:20.750562491 +0000 UTC m=+0.364799614,LastTimestamp:2026-07-28 20:08:20.751893657 +0000 UTC m=+0.366130782,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-243.ec2.internal,}"
Jul 28 20:08:20.784779 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.784762    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h5k6s"
Jul 28 20:08:20.796742 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.796728    2575 manager.go:337] "Starting Device Plugin manager"
Jul 28 20:08:20.796809 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.796754    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 20:08:20.796809 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.796763    2575 server.go:87] "Starting device plugin registration server"
Jul 28 20:08:20.796964 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.796954    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 20:08:20.797009 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.796965    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 20:08:20.797079 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.797066    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 20:08:20.797156 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.797131    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 20:08:20.797156 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.797141    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 20:08:20.797752 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.797737    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 20:08:20.797830 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.797775    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:20.800622 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.800567    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-243.ec2.internal.18c68cf86684d8e7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-243.ec2.internal,UID:ip-10-0-140-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-140-243.ec2.internal,},FirstTimestamp:2026-07-28 20:08:20.798740711 +0000 UTC m=+0.412977822,LastTimestamp:2026-07-28 20:08:20.798740711 +0000 UTC m=+0.412977822,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-243.ec2.internal,}"
Jul 28 20:08:20.827944 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.827921    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 20:08:20.833134 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.828989    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 20:08:20.833134 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.829034    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 20:08:20.833134 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.829051    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 20:08:20.833134 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.829079    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 20:08:20.840936 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.840921    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:08:20.898065 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.898011    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:08:20.898803 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.898790    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:08:20.898858 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.898817    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:08:20.898858 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.898831    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:08:20.898858 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.898857    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-243.ec2.internal"
Jul 28 20:08:20.906049 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.906035    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-243.ec2.internal"
Jul 28 20:08:20.906103 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.906055    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-243.ec2.internal\": node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:20.920672 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.920655    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:20.929471 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.929450    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal"]
Jul 28 20:08:20.929521 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.929505    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:08:20.930374 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.930360    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:08:20.930422 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.930388    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:08:20.930422 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.930399    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:08:20.932457 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.932439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee20295bd64b19c1c8203cd6fda0cfc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal\" (UID: \"ee20295bd64b19c1c8203cd6fda0cfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:20.932511 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.932484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee20295bd64b19c1c8203cd6fda0cfc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal\" (UID: \"ee20295bd64b19c1c8203cd6fda0cfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:20.932708 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.932694    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:08:20.932821 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.932807    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:20.934548 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.933006    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:08:20.934912 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.934896    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:08:20.934993 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.934927    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:08:20.934993 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.934942    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:08:20.934993 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.934961    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:08:20.934993 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.934987    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:08:20.935184 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.935001    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:08:20.937277 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.937262    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:20.937354 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.937286    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:08:20.938014 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.937996    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:08:20.938089 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.938024    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:08:20.938089 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:20.938033    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:08:20.956813 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.956793    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-243.ec2.internal\" not found" node="ip-10-0-140-243.ec2.internal"
Jul 28 20:08:20.961203 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:20.961189    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-243.ec2.internal\" not found" node="ip-10-0-140-243.ec2.internal"
Jul 28 20:08:21.020790 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:21.020772    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:21.032887 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.032872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee20295bd64b19c1c8203cd6fda0cfc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal\" (UID: \"ee20295bd64b19c1c8203cd6fda0cfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:21.032946 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.032893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee20295bd64b19c1c8203cd6fda0cfc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal\" (UID: \"ee20295bd64b19c1c8203cd6fda0cfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:21.032946 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.032914    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6bbf87afc187fb5805cf1fa6a52b872b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-243.ec2.internal\" (UID: \"6bbf87afc187fb5805cf1fa6a52b872b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:21.033009 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.032954    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee20295bd64b19c1c8203cd6fda0cfc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal\" (UID: \"ee20295bd64b19c1c8203cd6fda0cfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:21.033009 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.032970    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee20295bd64b19c1c8203cd6fda0cfc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal\" (UID: \"ee20295bd64b19c1c8203cd6fda0cfc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:21.120940 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:21.120912    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:21.133136 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.133116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6bbf87afc187fb5805cf1fa6a52b872b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-243.ec2.internal\" (UID: \"6bbf87afc187fb5805cf1fa6a52b872b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:21.133202 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.133174    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6bbf87afc187fb5805cf1fa6a52b872b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-243.ec2.internal\" (UID: \"6bbf87afc187fb5805cf1fa6a52b872b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:21.222169 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:21.222094    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:21.261489 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.261463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:21.265442 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.265418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:21.323069 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:21.323042    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:21.429665 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:21.429630    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:21.532731 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:21.532642    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:21.632999 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:21.632966    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:21.666587 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.666563    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 20:08:21.666993 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.666712    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 20:08:21.733039 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:21.733011    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:21.790313 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.790209    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 20:03:21 +0000 UTC" deadline="2028-03-06 18:06:55.338208016 +0000 UTC"
Jul 28 20:08:21.790313 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.790273    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14085h58m33.547939274s"
Jul 28 20:08:21.833110 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:21.833078    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:21.863120 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:21.863091    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbf87afc187fb5805cf1fa6a52b872b.slice/crio-6a5a38a1f39cccb73926b3849098ade62ee4bb8ded0ef3bdf77de56c6e44db4e WatchSource:0}: Error finding container 6a5a38a1f39cccb73926b3849098ade62ee4bb8ded0ef3bdf77de56c6e44db4e: Status 404 returned error can't find the container with id 6a5a38a1f39cccb73926b3849098ade62ee4bb8ded0ef3bdf77de56c6e44db4e
Jul 28 20:08:21.863645 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:21.863623    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee20295bd64b19c1c8203cd6fda0cfc0.slice/crio-270fba1b5fd9c5f3a715ec134311450e44a90911f362f0fcd661a1c7d4d1ca7f WatchSource:0}: Error finding container 270fba1b5fd9c5f3a715ec134311450e44a90911f362f0fcd661a1c7d4d1ca7f: Status 404 returned error can't find the container with id 270fba1b5fd9c5f3a715ec134311450e44a90911f362f0fcd661a1c7d4d1ca7f
Jul 28 20:08:21.868284 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.868257    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:08:21.925161 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.925124    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:08:21.926276 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.926257    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:08:21.926363 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.926290    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:08:21.926363 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:21.926301    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:08:21.933231 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:21.933214    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:22.033391 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.033363    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:22.070098 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.070036    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:08:22.134935 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.134911    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:22.148460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.148437    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:08:22.235483 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.235455    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-243.ec2.internal\" not found"
Jul 28 20:08:22.291360 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.291334    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:08:22.331483 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.331423    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:22.342805 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.342786    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 20:08:22.343652 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.343635    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal"
Jul 28 20:08:22.353220 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.353188    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 20:08:22.706944 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.706915    2575 apiserver.go:52] "Watching apiserver"
Jul 28 20:08:22.712200 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.712175    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 20:08:22.712640 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.712607    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-j2kp2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2","openshift-cluster-node-tuning-operator/tuned-hqlbv","openshift-multus/multus-additional-cni-plugins-tjdxd","openshift-multus/network-metrics-daemon-6kmfb","openshift-network-diagnostics/network-check-target-wqnw5","openshift-network-operator/iptables-alerter-xx2lz","kube-system/konnectivity-agent-d5z5k","kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal","openshift-dns/node-resolver-n87ww","openshift-image-registry/node-ca-4db9q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal","openshift-multus/multus-5b5lv","openshift-ovn-kubernetes/ovnkube-node-s9zbl"]
Jul 28 20:08:22.715212 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.715190    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:22.718139 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.717644    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.718139 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.717768    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j226p\""
Jul 28 20:08:22.718139 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.717879    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:08:22.718139 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.717882    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 20:08:22.724355 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.724331    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 20:08:22.724535 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.724513    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.725366 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.725117    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.725944 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.725885    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:22.727091 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.726151    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:22.727091 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.726355    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 20:08:22.727091 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.726720    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 20:08:22.727091 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.726829    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 20:08:22.727378 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.727196    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dhxpz\""
Jul 28 20:08:22.729328 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.728762    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 20:08:22.729328 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.729126    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hpzgr\""
Jul 28 20:08:22.729328 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.729155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:22.729328 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.729228    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:22.729328 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.729234    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 20:08:22.729646 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.729370    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:08:22.729863 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.729843    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 20:08:22.730107 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.730069    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 20:08:22.730928 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.730910    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7l6gm\""
Jul 28 20:08:22.731805 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.731671    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 20:08:22.732542 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.732520    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.734778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.734525    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 20:08:22.734778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.734540    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 20:08:22.734778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.734582    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8fwkt\""
Jul 28 20:08:22.735058 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.735040    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:22.735117 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.735098    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:22.735175 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.735045    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 20:08:22.735410 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.735392    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 20:08:22.735474 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.735409    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 20:08:22.735613 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.735117    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 20:08:22.737898 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.737867    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.739927 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.739907    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 20:08:22.740025 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.739963    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 20:08:22.740025 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.740016    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 20:08:22.740161 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.740131    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rrvs7\""
Jul 28 20:08:22.740947 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.740928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:22.741036 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.740979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshw9\" (UniqueName: \"kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9\") pod \"network-check-target-wqnw5\" (UID: \"503db72d-49cf-4f24-9b5a-3b06f54fbbfc\") " pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:22.741036 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741016    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-log-socket\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.741108 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-cni-netd\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.741108 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-ovn-node-metrics-cert\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.741108 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741098    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-run-k8s-cni-cncf-io\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.741282 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-var-lib-cni-multus\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.741282 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a32039b-4d29-4357-b275-6e2569d91483-iptables-alerter-script\") pod \"iptables-alerter-xx2lz\" (UID: \"9a32039b-4d29-4357-b275-6e2569d91483\") " pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:22.741282 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-ovnkube-script-lib\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.741282 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741203    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-system-cni-dir\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.741282 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741256    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/376809ed-3882-4375-9c40-2a004901780c-cni-binary-copy\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.741476 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741282    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-var-lib-kubelet\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.741476 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65lvf\" (UniqueName: \"kubernetes.io/projected/3ed50e77-948a-4cbd-ae6e-920b8c190c29-kube-api-access-65lvf\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:22.741476 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741359    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-kubelet\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.741476 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-run-systemd\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.741476 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5nl\" (UniqueName: \"kubernetes.io/projected/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-kube-api-access-hn5nl\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.741476 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-var-lib-cni-bin\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.741723 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741506    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-modprobe-d\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.741723 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741553    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/389d759c-8c3b-4055-aca8-aed66207f818-etc-tuned\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.741723 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741585    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-cni-bin\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.741723 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741611    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.741723 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-cnibin\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.741723 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-multus-socket-dir-parent\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.741723 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:22.741723 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741718    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-systemd-units\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-os-release\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdmc\" (UniqueName: \"kubernetes.io/projected/376809ed-3882-4375-9c40-2a004901780c-kube-api-access-mzdmc\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741808    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-os-release\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-host\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbnqn\" (UniqueName: \"kubernetes.io/projected/389d759c-8c3b-4055-aca8-aed66207f818-kube-api-access-dbnqn\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741906    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-system-cni-dir\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741943    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-ovnkube-config\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.741979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-sysctl-conf\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742004    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-slash\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-run-ovn\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-multus-cni-dir\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.742111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-run-netns\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742121    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-hostroot\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-run\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/389d759c-8c3b-4055-aca8-aed66207f818-tmp\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-sysconfig\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwwh5\" (UniqueName: \"kubernetes.io/projected/ac142701-e24f-4e3f-90d4-0367bbafa10c-kube-api-access-jwwh5\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742324    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-etc-openvswitch\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742361    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-cnibin\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742395    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-sysctl-d\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742424    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-var-lib-kubelet\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742456    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-kubernetes\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-systemd\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-sys\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac142701-e24f-4e3f-90d4-0367bbafa10c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742579    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-run-netns\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-node-log\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.742729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742644    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-multus-conf-dir\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742672    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac142701-e24f-4e3f-90d4-0367bbafa10c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6lns\" (UniqueName: \"kubernetes.io/projected/9a32039b-4d29-4357-b275-6e2569d91483-kube-api-access-x6lns\") pod \"iptables-alerter-xx2lz\" (UID: \"9a32039b-4d29-4357-b275-6e2569d91483\") " pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742723    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-lib-modules\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742760    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-run-openvswitch\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742792    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/376809ed-3882-4375-9c40-2a004901780c-multus-daemon-config\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742815    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-etc-kubernetes\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742834    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742838    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a32039b-4d29-4357-b275-6e2569d91483-host-slash\") pod \"iptables-alerter-xx2lz\" (UID: \"9a32039b-4d29-4357-b275-6e2569d91483\") " pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.742982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac142701-e24f-4e3f-90d4-0367bbafa10c-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.743013    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.743041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-var-lib-openvswitch\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.743069    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-env-overrides\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.743092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-run-multus-certs\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.743556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.743402    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 20:08:22.744493 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.743704    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cr9z2\""
Jul 28 20:08:22.744493 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.743766    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:22.745953 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.745925    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:22.746161 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.746141    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 20:08:22.746364 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.746347    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 20:08:22.746467 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.746393    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 20:08:22.746936 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.746713    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t2zdx\""
Jul 28 20:08:22.747541 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.747518    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 20:08:22.747952 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.747932    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 20:08:22.748628 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.747988    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 20:08:22.748628 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.748485    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5s7cb\""
Jul 28 20:08:22.765801 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.765758    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w9ld5"
Jul 28 20:08:22.771454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.771432    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w9ld5"
Jul 28 20:08:22.832681 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.832657    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 20:08:22.834835 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.834795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal" event={"ID":"ee20295bd64b19c1c8203cd6fda0cfc0","Type":"ContainerStarted","Data":"270fba1b5fd9c5f3a715ec134311450e44a90911f362f0fcd661a1c7d4d1ca7f"}
Jul 28 20:08:22.835989 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.835964    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal" event={"ID":"6bbf87afc187fb5805cf1fa6a52b872b","Type":"ContainerStarted","Data":"6a5a38a1f39cccb73926b3849098ade62ee4bb8ded0ef3bdf77de56c6e44db4e"}
Jul 28 20:08:22.844378 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-kubelet\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.844492 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-run-systemd\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.844492 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844419    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5nl\" (UniqueName: \"kubernetes.io/projected/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-kube-api-access-hn5nl\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.844492 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844444    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-var-lib-cni-bin\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.844492 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-modprobe-d\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.844690 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844491    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-run-systemd\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.844690 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/389d759c-8c3b-4055-aca8-aed66207f818-etc-tuned\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.844690 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-cni-bin\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.844690 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.844690 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844611    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-cnibin\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.844690 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844635    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-multus-socket-dir-parent\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.844690 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844651    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-modprobe-d\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844701    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-cni-bin\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-kubelet\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.844752    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844756    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844791    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-cnibin\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.844820    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs podName:3ed50e77-948a-4cbd-ae6e-920b8c190c29 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:23.344798257 +0000 UTC m=+2.959035386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs") pod "network-metrics-daemon-6kmfb" (UID: "3ed50e77-948a-4cbd-ae6e-920b8c190c29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844660    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844833    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844842    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-var-lib-cni-bin\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-systemd-units\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-os-release\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844895    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-multus-socket-dir-parent\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdmc\" (UniqueName: \"kubernetes.io/projected/376809ed-3882-4375-9c40-2a004901780c-kube-api-access-mzdmc\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.844986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.844947    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-systemd-units\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88ph\" (UniqueName: \"kubernetes.io/projected/72c191ee-908c-452b-a284-2e08a556b13b-kube-api-access-n88ph\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-os-release\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845061    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-os-release\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-host\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845100    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbnqn\" (UniqueName: \"kubernetes.io/projected/389d759c-8c3b-4055-aca8-aed66207f818-kube-api-access-dbnqn\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845104    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-host\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-system-cni-dir\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845163    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-os-release\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-ovnkube-config\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845290    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-system-cni-dir\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845392    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-device-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-sysctl-conf\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845461    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-slash\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-run-ovn\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-multus-cni-dir\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.845669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845551    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-run-netns\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845557    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-sysctl-conf\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-hostroot\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845586    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-run-ovn\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845631    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-hostroot\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845633    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-run-netns\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845672    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-slash\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845675    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/598f2d8e-b1f5-4e33-8dcf-30400b6357c1-konnectivity-ca\") pod \"konnectivity-agent-d5z5k\" (UID: \"598f2d8e-b1f5-4e33-8dcf-30400b6357c1\") " pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845692    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-multus-cni-dir\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845702    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-sys-fs\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845735    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-run\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-run\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845795    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/389d759c-8c3b-4055-aca8-aed66207f818-tmp\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845831    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-ovnkube-config\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845851    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a7f2ff-c2cc-4b29-8bf2-66861c7931bc-host\") pod \"node-ca-4db9q\" (UID: \"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc\") " pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-sysconfig\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwwh5\" (UniqueName: \"kubernetes.io/projected/ac142701-e24f-4e3f-90d4-0367bbafa10c-kube-api-access-jwwh5\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845931    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-etc-openvswitch\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.846531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845955    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a7f2ff-c2cc-4b29-8bf2-66861c7931bc-serviceca\") pod \"node-ca-4db9q\" (UID: \"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc\") " pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.845978    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/598f2d8e-b1f5-4e33-8dcf-30400b6357c1-agent-certs\") pod \"konnectivity-agent-d5z5k\" (UID: \"598f2d8e-b1f5-4e33-8dcf-30400b6357c1\") " pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-cnibin\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846039    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d953ada0-b545-41e5-901e-4b69b4d33bae-kubelet-config\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-etc-selinux\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846114    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-sysctl-d\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846153    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-var-lib-kubelet\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846178    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5stp\" (UniqueName: \"kubernetes.io/projected/88a7f2ff-c2cc-4b29-8bf2-66861c7931bc-kube-api-access-v5stp\") pod \"node-ca-4db9q\" (UID: \"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc\") " pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846233    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d953ada0-b545-41e5-901e-4b69b4d33bae-dbus\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846363    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-etc-openvswitch\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846413    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-sysctl-d\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-cnibin\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846505    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-var-lib-kubelet\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846564    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-sysconfig\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-socket-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846775    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-kubernetes\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.847454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-systemd\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846818    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-sys\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846843    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac142701-e24f-4e3f-90d4-0367bbafa10c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846890    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-run-netns\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-node-log\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.846967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-multus-conf-dir\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac142701-e24f-4e3f-90d4-0367bbafa10c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847066    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lns\" (UniqueName: \"kubernetes.io/projected/9a32039b-4d29-4357-b275-6e2569d91483-kube-api-access-x6lns\") pod \"iptables-alerter-xx2lz\" (UID: \"9a32039b-4d29-4357-b275-6e2569d91483\") " pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-lib-modules\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-run-openvswitch\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/376809ed-3882-4375-9c40-2a004901780c-multus-daemon-config\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-etc-kubernetes\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-registration-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a32039b-4d29-4357-b275-6e2569d91483-host-slash\") pod \"iptables-alerter-xx2lz\" (UID: \"9a32039b-4d29-4357-b275-6e2569d91483\") " pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847252    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac142701-e24f-4e3f-90d4-0367bbafa10c-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847325    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5fg5\" (UniqueName: \"kubernetes.io/projected/f52174fd-3b1a-465b-b0f4-4068deab6013-kube-api-access-v5fg5\") pod \"node-resolver-n87ww\" (UID: \"f52174fd-3b1a-465b-b0f4-4068deab6013\") " pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-var-lib-openvswitch\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847389    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-systemd\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847389    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-env-overrides\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-run-multus-certs\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847461    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f52174fd-3b1a-465b-b0f4-4068deab6013-hosts-file\") pod \"node-resolver-n87ww\" (UID: \"f52174fd-3b1a-465b-b0f4-4068deab6013\") " pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847491    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshw9\" (UniqueName: \"kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9\") pod \"network-check-target-wqnw5\" (UID: \"503db72d-49cf-4f24-9b5a-3b06f54fbbfc\") " pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-log-socket\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-cni-netd\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-ovn-node-metrics-cert\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-run-k8s-cni-cncf-io\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-var-lib-cni-multus\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a32039b-4d29-4357-b275-6e2569d91483-iptables-alerter-script\") pod \"iptables-alerter-xx2lz\" (UID: \"9a32039b-4d29-4357-b275-6e2569d91483\") " pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847661    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac142701-e24f-4e3f-90d4-0367bbafa10c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-ovnkube-script-lib\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847700    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-system-cni-dir\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/376809ed-3882-4375-9c40-2a004901780c-cni-binary-copy\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847746    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-sys\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-var-lib-kubelet\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65lvf\" (UniqueName: \"kubernetes.io/projected/3ed50e77-948a-4cbd-ae6e-920b8c190c29-kube-api-access-65lvf\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-env-overrides\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847841    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52174fd-3b1a-465b-b0f4-4068deab6013-tmp-dir\") pod \"node-resolver-n87ww\" (UID: \"f52174fd-3b1a-465b-b0f4-4068deab6013\") " pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.847970    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-system-cni-dir\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.848465    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a32039b-4d29-4357-b275-6e2569d91483-iptables-alerter-script\") pod \"iptables-alerter-xx2lz\" (UID: \"9a32039b-4d29-4357-b275-6e2569d91483\") " pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.848676    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/376809ed-3882-4375-9c40-2a004901780c-cni-binary-copy\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.848737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-var-lib-kubelet\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.848937    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-ovnkube-script-lib\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.848988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-run-netns\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849037    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-node-log\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849088    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-var-lib-openvswitch\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849133    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-etc-kubernetes\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849165    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac142701-e24f-4e3f-90d4-0367bbafa10c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849220    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-run-openvswitch\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849294    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-etc-kubernetes\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849509    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-run-multus-certs\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.849916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849740    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-log-socket\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.850584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849740    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac142701-e24f-4e3f-90d4-0367bbafa10c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.850584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849792    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.850584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-multus-conf-dir\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.850584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849806    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a32039b-4d29-4357-b275-6e2569d91483-host-slash\") pod \"iptables-alerter-xx2lz\" (UID: \"9a32039b-4d29-4357-b275-6e2569d91483\") " pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:22.850584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849867    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/376809ed-3882-4375-9c40-2a004901780c-multus-daemon-config\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.850584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.849984    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/389d759c-8c3b-4055-aca8-aed66207f818-lib-modules\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.850584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.850295    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac142701-e24f-4e3f-90d4-0367bbafa10c-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.850584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.850369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-run-k8s-cni-cncf-io\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.850584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.850409    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/376809ed-3882-4375-9c40-2a004901780c-host-var-lib-cni-multus\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.850584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.850451    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-host-cni-netd\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.851034 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.850763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/389d759c-8c3b-4055-aca8-aed66207f818-etc-tuned\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.852285 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.852258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/389d759c-8c3b-4055-aca8-aed66207f818-tmp\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.853479 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.853454    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-ovn-node-metrics-cert\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.854495 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.854470    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:08:22.854599 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.854498    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:08:22.854599 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.854512    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rshw9 for pod openshift-network-diagnostics/network-check-target-wqnw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:22.854599 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.854590    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9 podName:503db72d-49cf-4f24-9b5a-3b06f54fbbfc nodeName:}" failed. No retries permitted until 2026-07-28 20:08:23.35456126 +0000 UTC m=+2.968798376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rshw9" (UniqueName: "kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9") pod "network-check-target-wqnw5" (UID: "503db72d-49cf-4f24-9b5a-3b06f54fbbfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:22.855156 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.855132    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwwh5\" (UniqueName: \"kubernetes.io/projected/ac142701-e24f-4e3f-90d4-0367bbafa10c-kube-api-access-jwwh5\") pod \"multus-additional-cni-plugins-tjdxd\" (UID: \"ac142701-e24f-4e3f-90d4-0367bbafa10c\") " pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:22.855658 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.855612    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbnqn\" (UniqueName: \"kubernetes.io/projected/389d759c-8c3b-4055-aca8-aed66207f818-kube-api-access-dbnqn\") pod \"tuned-hqlbv\" (UID: \"389d759c-8c3b-4055-aca8-aed66207f818\") " pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:22.856935 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.856910    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65lvf\" (UniqueName: \"kubernetes.io/projected/3ed50e77-948a-4cbd-ae6e-920b8c190c29-kube-api-access-65lvf\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:22.857251 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.857224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6lns\" (UniqueName: \"kubernetes.io/projected/9a32039b-4d29-4357-b275-6e2569d91483-kube-api-access-x6lns\") pod \"iptables-alerter-xx2lz\" (UID: \"9a32039b-4d29-4357-b275-6e2569d91483\") " pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:22.857722 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.857701    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5nl\" (UniqueName: \"kubernetes.io/projected/fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0-kube-api-access-hn5nl\") pod \"ovnkube-node-s9zbl\" (UID: \"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:22.857903 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.857885    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdmc\" (UniqueName: \"kubernetes.io/projected/376809ed-3882-4375-9c40-2a004901780c-kube-api-access-mzdmc\") pod \"multus-5b5lv\" (UID: \"376809ed-3882-4375-9c40-2a004901780c\") " pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:22.948119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a7f2ff-c2cc-4b29-8bf2-66861c7931bc-serviceca\") pod \"node-ca-4db9q\" (UID: \"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc\") " pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:22.948119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948120    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/598f2d8e-b1f5-4e33-8dcf-30400b6357c1-agent-certs\") pod \"konnectivity-agent-d5z5k\" (UID: \"598f2d8e-b1f5-4e33-8dcf-30400b6357c1\") " pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:22.948382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d953ada0-b545-41e5-901e-4b69b4d33bae-kubelet-config\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:22.948382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948188    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-etc-selinux\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5stp\" (UniqueName: \"kubernetes.io/projected/88a7f2ff-c2cc-4b29-8bf2-66861c7931bc-kube-api-access-v5stp\") pod \"node-ca-4db9q\" (UID: \"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc\") " pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:22.948382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d953ada0-b545-41e5-901e-4b69b4d33bae-dbus\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:22.948382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948261    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-socket-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-registration-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5fg5\" (UniqueName: \"kubernetes.io/projected/f52174fd-3b1a-465b-b0f4-4068deab6013-kube-api-access-v5fg5\") pod \"node-resolver-n87ww\" (UID: \"f52174fd-3b1a-465b-b0f4-4068deab6013\") " pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:22.948382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948327    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f52174fd-3b1a-465b-b0f4-4068deab6013-hosts-file\") pod \"node-resolver-n87ww\" (UID: \"f52174fd-3b1a-465b-b0f4-4068deab6013\") " pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948414    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d953ada0-b545-41e5-901e-4b69b4d33bae-dbus\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948427    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-etc-selinux\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d953ada0-b545-41e5-901e-4b69b4d33bae-kubelet-config\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52174fd-3b1a-465b-b0f4-4068deab6013-tmp-dir\") pod \"node-resolver-n87ww\" (UID: \"f52174fd-3b1a-465b-b0f4-4068deab6013\") " pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948526    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88ph\" (UniqueName: \"kubernetes.io/projected/72c191ee-908c-452b-a284-2e08a556b13b-kube-api-access-n88ph\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948606    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-device-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948614    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-socket-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/598f2d8e-b1f5-4e33-8dcf-30400b6357c1-konnectivity-ca\") pod \"konnectivity-agent-d5z5k\" (UID: \"598f2d8e-b1f5-4e33-8dcf-30400b6357c1\") " pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-sys-fs\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a7f2ff-c2cc-4b29-8bf2-66861c7931bc-host\") pod \"node-ca-4db9q\" (UID: \"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc\") " pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a7f2ff-c2cc-4b29-8bf2-66861c7931bc-host\") pod \"node-ca-4db9q\" (UID: \"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc\") " pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948827    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-registration-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.948847 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.948843    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:22.949550 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:22.948904    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret podName:d953ada0-b545-41e5-901e-4b69b4d33bae nodeName:}" failed. No retries permitted until 2026-07-28 20:08:23.448885975 +0000 UTC m=+3.063123088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret") pod "global-pull-secret-syncer-j2kp2" (UID: "d953ada0-b545-41e5-901e-4b69b4d33bae") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:22.949550 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948961    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-device-dir\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.949550 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.948971    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a7f2ff-c2cc-4b29-8bf2-66861c7931bc-serviceca\") pod \"node-ca-4db9q\" (UID: \"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc\") " pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:22.949550 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.949117    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52174fd-3b1a-465b-b0f4-4068deab6013-tmp-dir\") pod \"node-resolver-n87ww\" (UID: \"f52174fd-3b1a-465b-b0f4-4068deab6013\") " pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:22.949550 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.949193    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72c191ee-908c-452b-a284-2e08a556b13b-sys-fs\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.949550 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.949263    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f52174fd-3b1a-465b-b0f4-4068deab6013-hosts-file\") pod \"node-resolver-n87ww\" (UID: \"f52174fd-3b1a-465b-b0f4-4068deab6013\") " pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:22.949550 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.949488    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/598f2d8e-b1f5-4e33-8dcf-30400b6357c1-konnectivity-ca\") pod \"konnectivity-agent-d5z5k\" (UID: \"598f2d8e-b1f5-4e33-8dcf-30400b6357c1\") " pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:22.951221 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.951180    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/598f2d8e-b1f5-4e33-8dcf-30400b6357c1-agent-certs\") pod \"konnectivity-agent-d5z5k\" (UID: \"598f2d8e-b1f5-4e33-8dcf-30400b6357c1\") " pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:22.958300 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.958238    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5fg5\" (UniqueName: \"kubernetes.io/projected/f52174fd-3b1a-465b-b0f4-4068deab6013-kube-api-access-v5fg5\") pod \"node-resolver-n87ww\" (UID: \"f52174fd-3b1a-465b-b0f4-4068deab6013\") " pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:22.958485 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.958468    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88ph\" (UniqueName: \"kubernetes.io/projected/72c191ee-908c-452b-a284-2e08a556b13b-kube-api-access-n88ph\") pod \"aws-ebs-csi-driver-node-q8qw2\" (UID: \"72c191ee-908c-452b-a284-2e08a556b13b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:22.963472 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.963447    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5stp\" (UniqueName: \"kubernetes.io/projected/88a7f2ff-c2cc-4b29-8bf2-66861c7931bc-kube-api-access-v5stp\") pod \"node-ca-4db9q\" (UID: \"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc\") " pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:22.976516 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:22.976495    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:08:23.038655 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.038612    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xx2lz"
Jul 28 20:08:23.047538 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.047514    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5b5lv"
Jul 28 20:08:23.057432 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.057410    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hqlbv"
Jul 28 20:08:23.064432 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.064406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tjdxd"
Jul 28 20:08:23.073313 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.073293    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:23.084119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.084098    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2"
Jul 28 20:08:23.092137 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.092113    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n87ww"
Jul 28 20:08:23.101177 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.101157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4db9q"
Jul 28 20:08:23.105942 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.105926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:23.351758 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.351662    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:23.351918 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:23.351868    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:23.351969 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:23.351955    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs podName:3ed50e77-948a-4cbd-ae6e-920b8c190c29 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:24.351931694 +0000 UTC m=+3.966168824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs") pod "network-metrics-daemon-6kmfb" (UID: "3ed50e77-948a-4cbd-ae6e-920b8c190c29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:23.452809 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.452774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshw9\" (UniqueName: \"kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9\") pod \"network-check-target-wqnw5\" (UID: \"503db72d-49cf-4f24-9b5a-3b06f54fbbfc\") " pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:23.453023 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.452846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:23.453023 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:23.452960    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:23.453023 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:23.452985    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:08:23.453023 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:23.453012    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:08:23.453023 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:23.453026    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rshw9 for pod openshift-network-diagnostics/network-check-target-wqnw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:23.453303 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:23.453035    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret podName:d953ada0-b545-41e5-901e-4b69b4d33bae nodeName:}" failed. No retries permitted until 2026-07-28 20:08:24.453015795 +0000 UTC m=+4.067252929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret") pod "global-pull-secret-syncer-j2kp2" (UID: "d953ada0-b545-41e5-901e-4b69b4d33bae") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:23.453303 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:23.453081    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9 podName:503db72d-49cf-4f24-9b5a-3b06f54fbbfc nodeName:}" failed. No retries permitted until 2026-07-28 20:08:24.453064473 +0000 UTC m=+4.067301593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshw9" (UniqueName: "kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9") pod "network-check-target-wqnw5" (UID: "503db72d-49cf-4f24-9b5a-3b06f54fbbfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:23.775501 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.775445    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 20:03:22 +0000 UTC" deadline="2028-01-30 04:17:19.663052003 +0000 UTC"
Jul 28 20:08:23.775501 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:23.775495    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13208h8m55.887562035s"
Jul 28 20:08:23.944779 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:23.944750    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a7f2ff_c2cc_4b29_8bf2_66861c7931bc.slice/crio-ce3d9b61083f09af131172bc6c734cd492b18b38848a0de77cf120fe2bd48a3a WatchSource:0}: Error finding container ce3d9b61083f09af131172bc6c734cd492b18b38848a0de77cf120fe2bd48a3a: Status 404 returned error can't find the container with id ce3d9b61083f09af131172bc6c734cd492b18b38848a0de77cf120fe2bd48a3a
Jul 28 20:08:23.945954 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:23.945883    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac142701_e24f_4e3f_90d4_0367bbafa10c.slice/crio-e691d9acaee070da2efbb13b150b0d99693c7fcd7aef6a3ade47cf84105d8995 WatchSource:0}: Error finding container e691d9acaee070da2efbb13b150b0d99693c7fcd7aef6a3ade47cf84105d8995: Status 404 returned error can't find the container with id e691d9acaee070da2efbb13b150b0d99693c7fcd7aef6a3ade47cf84105d8995
Jul 28 20:08:23.955026 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:23.955001    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a32039b_4d29_4357_b275_6e2569d91483.slice/crio-6db15eeccf37a3ba44905521a57045775dfd9f91d402940d3a7c9c5389ff20ed WatchSource:0}: Error finding container 6db15eeccf37a3ba44905521a57045775dfd9f91d402940d3a7c9c5389ff20ed: Status 404 returned error can't find the container with id 6db15eeccf37a3ba44905521a57045775dfd9f91d402940d3a7c9c5389ff20ed
Jul 28 20:08:24.360523 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.359687    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:24.360523 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.359930    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:24.360523 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.360082    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs podName:3ed50e77-948a-4cbd-ae6e-920b8c190c29 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:26.360060354 +0000 UTC m=+5.974297470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs") pod "network-metrics-daemon-6kmfb" (UID: "3ed50e77-948a-4cbd-ae6e-920b8c190c29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:24.461274 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.460415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshw9\" (UniqueName: \"kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9\") pod \"network-check-target-wqnw5\" (UID: \"503db72d-49cf-4f24-9b5a-3b06f54fbbfc\") " pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:24.461274 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.460495    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:24.461274 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.460624    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:24.461274 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.460683    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret podName:d953ada0-b545-41e5-901e-4b69b4d33bae nodeName:}" failed. No retries permitted until 2026-07-28 20:08:26.460666166 +0000 UTC m=+6.074903283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret") pod "global-pull-secret-syncer-j2kp2" (UID: "d953ada0-b545-41e5-901e-4b69b4d33bae") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:24.461274 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.461145    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:08:24.461274 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.461164    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:08:24.461274 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.461176    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rshw9 for pod openshift-network-diagnostics/network-check-target-wqnw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:24.461274 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.461239    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9 podName:503db72d-49cf-4f24-9b5a-3b06f54fbbfc nodeName:}" failed. No retries permitted until 2026-07-28 20:08:26.461206721 +0000 UTC m=+6.075443834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshw9" (UniqueName: "kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9") pod "network-check-target-wqnw5" (UID: "503db72d-49cf-4f24-9b5a-3b06f54fbbfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:24.776032 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.775990    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 20:03:22 +0000 UTC" deadline="2028-03-10 17:07:57.38875192 +0000 UTC"
Jul 28 20:08:24.776032 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.776029    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14180h59m32.612727193s"
Jul 28 20:08:24.831078 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.830371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:24.831078 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.830406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:24.831078 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.830407    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:24.831078 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.830517    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:24.831078 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.830952    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:24.831078 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:24.831039    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:24.861273 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.861096    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2" event={"ID":"72c191ee-908c-452b-a284-2e08a556b13b","Type":"ContainerStarted","Data":"bd11e25eb696da5bda86927aa79113f2e1eed5fce9d40f3247655bfa7fd3c9b7"}
Jul 28 20:08:24.871933 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.871840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjdxd" event={"ID":"ac142701-e24f-4e3f-90d4-0367bbafa10c","Type":"ContainerStarted","Data":"e691d9acaee070da2efbb13b150b0d99693c7fcd7aef6a3ade47cf84105d8995"}
Jul 28 20:08:24.880862 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.880680    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4db9q" event={"ID":"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc","Type":"ContainerStarted","Data":"ce3d9b61083f09af131172bc6c734cd492b18b38848a0de77cf120fe2bd48a3a"}
Jul 28 20:08:24.892739 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.892706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl" event={"ID":"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0","Type":"ContainerStarted","Data":"0e15430b7e46f91de975d7ad5b0b5f00c870ae54db26a6e9ec28bf09f3430da7"}
Jul 28 20:08:24.899526 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.899494    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal" event={"ID":"6bbf87afc187fb5805cf1fa6a52b872b","Type":"ContainerStarted","Data":"a30e881d15d720d9cb593bdc3d14b154c0b67f18ae01a745ea53f557cc6a9c77"}
Jul 28 20:08:24.908798 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.908769    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d5z5k" event={"ID":"598f2d8e-b1f5-4e33-8dcf-30400b6357c1","Type":"ContainerStarted","Data":"ea3133ac57a815cfe372ac09e3daeaaea1ebe98a5ad84d654fb0f6582972b8bd"}
Jul 28 20:08:24.911509 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.911483    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5b5lv" event={"ID":"376809ed-3882-4375-9c40-2a004901780c","Type":"ContainerStarted","Data":"fffaa38b14bc7211de8aa1312ca4675b3ac627b46400065cbd5a521a9e586f20"}
Jul 28 20:08:24.930125 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.930093    2575 generic.go:356] "Generic (PLEG): container finished" podID="ee20295bd64b19c1c8203cd6fda0cfc0" containerID="13324844bfe28f1c425b4351745271ae42889d892fce1b8ffe51825976230364" exitCode=0
Jul 28 20:08:24.930293 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.930190    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal" event={"ID":"ee20295bd64b19c1c8203cd6fda0cfc0","Type":"ContainerDied","Data":"13324844bfe28f1c425b4351745271ae42889d892fce1b8ffe51825976230364"}
Jul 28 20:08:24.953599 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.945445    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xx2lz" event={"ID":"9a32039b-4d29-4357-b275-6e2569d91483","Type":"ContainerStarted","Data":"6db15eeccf37a3ba44905521a57045775dfd9f91d402940d3a7c9c5389ff20ed"}
Jul 28 20:08:24.953599 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.947585    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hqlbv" event={"ID":"389d759c-8c3b-4055-aca8-aed66207f818","Type":"ContainerStarted","Data":"6e65eb2854cd16085394d0b494150dee8c34c009213c77e7697e8260c3013854"}
Jul 28 20:08:24.955363 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.955007    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-243.ec2.internal" podStartSLOduration=2.95493335 podStartE2EDuration="2.95493335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:24.92316957 +0000 UTC m=+4.537406719" watchObservedRunningTime="2026-07-28 20:08:24.95493335 +0000 UTC m=+4.569170485"
Jul 28 20:08:24.964654 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:24.964626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n87ww" event={"ID":"f52174fd-3b1a-465b-b0f4-4068deab6013","Type":"ContainerStarted","Data":"8a7be0760b5ac00123e6aa16e5e71fc01c771eb82b5a0e9a0069fd8a3796f27e"}
Jul 28 20:08:25.987563 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:25.987530    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal" event={"ID":"ee20295bd64b19c1c8203cd6fda0cfc0","Type":"ContainerStarted","Data":"ac5d3edf07637fecaaaf20408b17dea2b362b7c61b3b3bd6abadbb164bd1e36d"}
Jul 28 20:08:26.373969 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:26.373879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:26.374129 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.374096    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:26.374188 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.374159    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs podName:3ed50e77-948a-4cbd-ae6e-920b8c190c29 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:30.374140368 +0000 UTC m=+9.988377485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs") pod "network-metrics-daemon-6kmfb" (UID: "3ed50e77-948a-4cbd-ae6e-920b8c190c29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:26.474714 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:26.474666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshw9\" (UniqueName: \"kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9\") pod \"network-check-target-wqnw5\" (UID: \"503db72d-49cf-4f24-9b5a-3b06f54fbbfc\") " pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:26.474897 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:26.474754    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:26.474965 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.474908    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:26.475022 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.474968    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret podName:d953ada0-b545-41e5-901e-4b69b4d33bae nodeName:}" failed. No retries permitted until 2026-07-28 20:08:30.474950853 +0000 UTC m=+10.089187973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret") pod "global-pull-secret-syncer-j2kp2" (UID: "d953ada0-b545-41e5-901e-4b69b4d33bae") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:26.477182 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.475121    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:08:26.477182 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.475191    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:08:26.477182 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.475203    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rshw9 for pod openshift-network-diagnostics/network-check-target-wqnw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:26.477182 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.475264    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9 podName:503db72d-49cf-4f24-9b5a-3b06f54fbbfc nodeName:}" failed. No retries permitted until 2026-07-28 20:08:30.475248751 +0000 UTC m=+10.089485868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshw9" (UniqueName: "kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9") pod "network-check-target-wqnw5" (UID: "503db72d-49cf-4f24-9b5a-3b06f54fbbfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:26.832218 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:26.831439    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:26.832218 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:26.831439    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:26.832218 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.831580    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:26.832218 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.831621    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:26.832218 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:26.831439    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:26.832218 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:26.831714    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:28.840750 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:28.840091    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:28.840750 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:28.840294    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:28.840750 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:28.840366    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:28.840750 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:28.840510    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:28.840750 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:28.840578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:28.840750 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:28.840682    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:30.411520 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:30.411486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:30.411891 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.411704    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:30.411891 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.411782    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs podName:3ed50e77-948a-4cbd-ae6e-920b8c190c29 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:38.411758546 +0000 UTC m=+18.025995661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs") pod "network-metrics-daemon-6kmfb" (UID: "3ed50e77-948a-4cbd-ae6e-920b8c190c29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:30.512635 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:30.512595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshw9\" (UniqueName: \"kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9\") pod \"network-check-target-wqnw5\" (UID: \"503db72d-49cf-4f24-9b5a-3b06f54fbbfc\") " pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:30.512804 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:30.512656    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:30.512804 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.512779    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:30.512924 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.512843    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret podName:d953ada0-b545-41e5-901e-4b69b4d33bae nodeName:}" failed. No retries permitted until 2026-07-28 20:08:38.512822715 +0000 UTC m=+18.127059830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret") pod "global-pull-secret-syncer-j2kp2" (UID: "d953ada0-b545-41e5-901e-4b69b4d33bae") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:30.513292 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.513271    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:08:30.513384 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.513298    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:08:30.513384 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.513308    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rshw9 for pod openshift-network-diagnostics/network-check-target-wqnw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:30.513384 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.513344    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9 podName:503db72d-49cf-4f24-9b5a-3b06f54fbbfc nodeName:}" failed. No retries permitted until 2026-07-28 20:08:38.513331767 +0000 UTC m=+18.127568878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshw9" (UniqueName: "kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9") pod "network-check-target-wqnw5" (UID: "503db72d-49cf-4f24-9b5a-3b06f54fbbfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:30.831451 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:30.830329    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:30.831451 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:30.830329    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:30.831451 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:30.830461    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:30.831451 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.830456    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:30.831451 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.830525    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:30.831451 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:30.830614    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:32.830071 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:32.830036    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:32.830613 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:32.830171    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:32.830613 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:32.830587    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:32.830716 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:32.830695    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:32.830767 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:32.830752    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:32.830842 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:32.830820    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:34.829652 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:34.829615    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:34.830020 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:34.829624    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:34.830020 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:34.829733    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:34.830020 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:34.829892    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:34.830020 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:34.829914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:34.830020 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:34.829984    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:36.829550 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:36.829515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:36.829550 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:36.829550    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:36.830026 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:36.829515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:36.830026 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:36.829635    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:36.830026 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:36.829723    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:36.830026 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:36.829793    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:38.474662 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:38.474621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:38.475085 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.474776    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:38.475085 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.474847    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs podName:3ed50e77-948a-4cbd-ae6e-920b8c190c29 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:54.474831216 +0000 UTC m=+34.089068327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs") pod "network-metrics-daemon-6kmfb" (UID: "3ed50e77-948a-4cbd-ae6e-920b8c190c29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:38.575979 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:38.575946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshw9\" (UniqueName: \"kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9\") pod \"network-check-target-wqnw5\" (UID: \"503db72d-49cf-4f24-9b5a-3b06f54fbbfc\") " pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:38.576164 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:38.576141    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:38.576213 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.576148    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:08:38.576213 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.576199    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:38.576347 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.576217    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:08:38.576347 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.576253    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rshw9 for pod openshift-network-diagnostics/network-check-target-wqnw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:38.576347 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.576284    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret podName:d953ada0-b545-41e5-901e-4b69b4d33bae nodeName:}" failed. No retries permitted until 2026-07-28 20:08:54.576267037 +0000 UTC m=+34.190504150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret") pod "global-pull-secret-syncer-j2kp2" (UID: "d953ada0-b545-41e5-901e-4b69b4d33bae") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:38.576347 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.576305    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9 podName:503db72d-49cf-4f24-9b5a-3b06f54fbbfc nodeName:}" failed. No retries permitted until 2026-07-28 20:08:54.576293079 +0000 UTC m=+34.190530207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshw9" (UniqueName: "kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9") pod "network-check-target-wqnw5" (UID: "503db72d-49cf-4f24-9b5a-3b06f54fbbfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:38.830202 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:38.830096    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:38.830202 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:38.830098    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:38.830438 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.830238    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:38.830438 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.830316    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:38.830438 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:38.830103    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:38.830438 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:38.830390    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:40.830980 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:40.830953    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:40.831413 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:40.831051    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:40.831413 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:40.831141    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:40.831413 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:40.831278    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:40.831413 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:40.831342    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:40.831603 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:40.831415    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:42.830390 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:42.830360    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:42.830785 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:42.830508    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:42.830785 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:42.830536    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:42.830785 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:42.830625    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:42.830785 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:42.830674    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:42.830785 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:42.830744    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:43.021444 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.021102    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n87ww" event={"ID":"f52174fd-3b1a-465b-b0f4-4068deab6013","Type":"ContainerStarted","Data":"146a6334d3840349d6315ca173484e76492f729601b5366dc396cdeba10dc775"}
Jul 28 20:08:43.022744 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.022721    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2" event={"ID":"72c191ee-908c-452b-a284-2e08a556b13b","Type":"ContainerStarted","Data":"c53969ff79c9e55d811d4a147fbdf3c4a38cfd64ff98fbb6493afb1ca57cf727"}
Jul 28 20:08:43.024158 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.024041    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac142701-e24f-4e3f-90d4-0367bbafa10c" containerID="de0cca04e4aedbdff1b1a863ec5e96a7f67e7db5ede6eb85184d0b23f6c6677f" exitCode=0
Jul 28 20:08:43.024158 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.024124    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjdxd" event={"ID":"ac142701-e24f-4e3f-90d4-0367bbafa10c","Type":"ContainerDied","Data":"de0cca04e4aedbdff1b1a863ec5e96a7f67e7db5ede6eb85184d0b23f6c6677f"}
Jul 28 20:08:43.025622 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.025602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4db9q" event={"ID":"88a7f2ff-c2cc-4b29-8bf2-66861c7931bc","Type":"ContainerStarted","Data":"1b82eb96bb10267758ed0a0f1a8664d3a41be09af90b1a2cf9aaf20dd2331551"}
Jul 28 20:08:43.028948 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.028810    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl" event={"ID":"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0","Type":"ContainerStarted","Data":"3625893b575c5803a86c2f9610303d7434b72bc97b01776ed8151e7400f0f819"}
Jul 28 20:08:43.028948 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.028892    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl" event={"ID":"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0","Type":"ContainerStarted","Data":"6114f4fd87aee159f4e8fc27d3bd9e121ff3c174a89b91198b382e6097178258"}
Jul 28 20:08:43.030373 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.030329    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d5z5k" event={"ID":"598f2d8e-b1f5-4e33-8dcf-30400b6357c1","Type":"ContainerStarted","Data":"cfbc148ee4b3d253d021540744d079cbebc402bc55716b1183779c9b02ad05d0"}
Jul 28 20:08:43.032091 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.032070    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5b5lv" event={"ID":"376809ed-3882-4375-9c40-2a004901780c","Type":"ContainerStarted","Data":"adec8f8bd7c760de3db801deca67cd07bc141c78def244191ed805fdbc61f923"}
Jul 28 20:08:43.033429 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.033407    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hqlbv" event={"ID":"389d759c-8c3b-4055-aca8-aed66207f818","Type":"ContainerStarted","Data":"de586da7e9b4b54e52e17f445911c92c6ff8c49bccb07f4c24e34b78f41a31f6"}
Jul 28 20:08:43.034529 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.034496    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-243.ec2.internal" podStartSLOduration=21.034469354 podStartE2EDuration="21.034469354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:26.002485274 +0000 UTC m=+5.616722407" watchObservedRunningTime="2026-07-28 20:08:43.034469354 +0000 UTC m=+22.648706480"
Jul 28 20:08:43.035256 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.035210    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n87ww" podStartSLOduration=3.261213069 podStartE2EDuration="22.035199633s" totalImagesPullingTime="18.773986564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:43.034694409 +0000 UTC m=+22.648931543" watchObservedRunningTime="2026-07-28 20:08:43.035199633 +0000 UTC m=+22.649436765"
Jul 28 20:08:43.068630 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.068588    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4db9q" podStartSLOduration=3.290564868 podStartE2EDuration="22.068573215s" totalImagesPullingTime="18.778008347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:43.068419243 +0000 UTC m=+22.682656377" watchObservedRunningTime="2026-07-28 20:08:43.068573215 +0000 UTC m=+22.682810347"
Jul 28 20:08:43.084266 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.084193    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5b5lv" podStartSLOduration=3.248490603 podStartE2EDuration="22.084177505s" totalImagesPullingTime="18.835686902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:43.083878297 +0000 UTC m=+22.698115446" watchObservedRunningTime="2026-07-28 20:08:43.084177505 +0000 UTC m=+22.698414635"
Jul 28 20:08:43.103716 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.103674    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hqlbv" podStartSLOduration=3.311636287 podStartE2EDuration="22.103660367s" totalImagesPullingTime="18.79202408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:43.10333218 +0000 UTC m=+22.717569313" watchObservedRunningTime="2026-07-28 20:08:43.103660367 +0000 UTC m=+22.717897500"
Jul 28 20:08:43.120775 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:43.120666    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d5z5k" podStartSLOduration=7.552682807 podStartE2EDuration="22.120650435s" totalImagesPullingTime="14.567967628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:43.120268726 +0000 UTC m=+22.734505859" watchObservedRunningTime="2026-07-28 20:08:43.120650435 +0000 UTC m=+22.734887567"
Jul 28 20:08:44.039605 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.039528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl" event={"ID":"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0","Type":"ContainerStarted","Data":"211a587c68afa60844fb7307dc476955c86eff4fbeb7d57d02ba5f4c9796352c"}
Jul 28 20:08:44.040438 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.039653    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl" event={"ID":"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0","Type":"ContainerStarted","Data":"c2d1ab8a4bc8556caf2fe770dee479e6f9435c74574a5bd78e1d1d7d9026eef7"}
Jul 28 20:08:44.040438 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.039686    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl" event={"ID":"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0","Type":"ContainerStarted","Data":"228b7b6d0c51592736578dd7cb6cc1c5f63d28270b04016602c11d26f87dea03"}
Jul 28 20:08:44.040438 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.039721    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl" event={"ID":"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0","Type":"ContainerStarted","Data":"a39ea3745ef2e14ffb1aa5a6d10ac17d66639a9b95f9c964b3777669ee44f6f5"}
Jul 28 20:08:44.041009 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.040982    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xx2lz" event={"ID":"9a32039b-4d29-4357-b275-6e2569d91483","Type":"ContainerStarted","Data":"50e46f8295ffcdf34cce5ecde5995477624fe8db79d9cd0cd349b9989dafe936"}
Jul 28 20:08:44.054896 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.054850    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xx2lz" podStartSLOduration=4.286618173 podStartE2EDuration="23.054834634s" totalImagesPullingTime="18.768216461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:44.054113331 +0000 UTC m=+23.668350465" watchObservedRunningTime="2026-07-28 20:08:44.054834634 +0000 UTC m=+23.669071767"
Jul 28 20:08:44.282109 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.282069    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 20:08:44.808331 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.808197    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T20:08:44.28209216Z","UUID":"636353ee-d49b-44fc-81b3-835b42c305e5","Handler":null,"Name":"","Endpoint":""}
Jul 28 20:08:44.810113 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.810089    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 20:08:44.810255 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.810134    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 20:08:44.830030 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.829994    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:44.830206 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:44.830096    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:44.830418 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.830401    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:44.830525 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:44.830502    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:44.830591 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:44.830581    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:44.830669 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:44.830647    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:45.045825 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:45.045791    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2" event={"ID":"72c191ee-908c-452b-a284-2e08a556b13b","Type":"ContainerStarted","Data":"6713382d941fc8e5ec8a86708dbb22c6b3887e99a2263e05fa28ce893b651a03"}
Jul 28 20:08:46.023528 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:46.023485    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:46.024126 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:46.024103    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:46.050283 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:46.050243    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2" event={"ID":"72c191ee-908c-452b-a284-2e08a556b13b","Type":"ContainerStarted","Data":"c108b899dc537823f662ad1aa10bf85c868c7eaf84a8e749e8e2fe94555d5255"}
Jul 28 20:08:46.050718 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:46.050460    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:46.050975 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:46.050947    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d5z5k"
Jul 28 20:08:46.070020 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:46.069972    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q8qw2" podStartSLOduration=4.112499508 podStartE2EDuration="25.06995596s" totalImagesPullingTime="20.957456452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:46.069625041 +0000 UTC m=+25.683862206" watchObservedRunningTime="2026-07-28 20:08:46.06995596 +0000 UTC m=+25.684193092"
Jul 28 20:08:46.833406 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:46.833246    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:46.833600 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:46.833285    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:46.833600 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:46.833502    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:46.833600 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:46.833303    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:46.833600 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:46.833572    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:46.833801 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:46.833683    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:47.055452 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:47.055421    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl" event={"ID":"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0","Type":"ContainerStarted","Data":"2794dd21254c34b0127d6821752150311f027b997f484ea1477facf6987cd6e9"}
Jul 28 20:08:48.059152 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:48.059121    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac142701-e24f-4e3f-90d4-0367bbafa10c" containerID="325605126e9170b62a0b722a208e6fe6d103fcc436b5c50cb90b39d757d89d23" exitCode=0
Jul 28 20:08:48.059833 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:48.059202    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjdxd" event={"ID":"ac142701-e24f-4e3f-90d4-0367bbafa10c","Type":"ContainerDied","Data":"325605126e9170b62a0b722a208e6fe6d103fcc436b5c50cb90b39d757d89d23"}
Jul 28 20:08:48.830020 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:48.829985    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:48.830209 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:48.829986    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:48.830209 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:48.830118    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:48.830209 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:48.830158    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:48.830209 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:48.829985    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:48.830437 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:48.830221    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:49.063396 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:49.063363    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac142701-e24f-4e3f-90d4-0367bbafa10c" containerID="8b7d45b7f3ddb156b0fe3a16ec41e56f333195cb45aed49ffc91fe14f41f4d68" exitCode=0
Jul 28 20:08:49.063949 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:49.063443    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjdxd" event={"ID":"ac142701-e24f-4e3f-90d4-0367bbafa10c","Type":"ContainerDied","Data":"8b7d45b7f3ddb156b0fe3a16ec41e56f333195cb45aed49ffc91fe14f41f4d68"}
Jul 28 20:08:50.068909 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:50.068716    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl" event={"ID":"fe7a7c46-41c4-4e2f-a653-8300ff6bbdb0","Type":"ContainerStarted","Data":"525e6d6e803c2dcf261bbc5e2e235fb75f76f9dc48418829f9862bb6aa295951"}
Jul 28 20:08:50.069447 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:50.068974    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:50.069447 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:50.069022    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:50.084514 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:50.084485    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:50.095594 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:50.095556    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl" podStartSLOduration=10.234356114 podStartE2EDuration="29.095541569s" totalImagesPullingTime="18.861185455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:50.094262356 +0000 UTC m=+29.708499491" watchObservedRunningTime="2026-07-28 20:08:50.095541569 +0000 UTC m=+29.709778701"
Jul 28 20:08:50.830002 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:50.829975    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:50.830174 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:50.830074    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:50.830174 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:50.830075    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:50.830174 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:50.830159    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:50.830174 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:50.830170    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:50.830419 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:50.830381    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:51.073869 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:51.073830    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac142701-e24f-4e3f-90d4-0367bbafa10c" containerID="f7cd82e673ff4568b0655f95b3490814da8170afd7e7e7b599161c965aea7eaa" exitCode=0
Jul 28 20:08:51.074297 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:51.073920    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjdxd" event={"ID":"ac142701-e24f-4e3f-90d4-0367bbafa10c","Type":"ContainerDied","Data":"f7cd82e673ff4568b0655f95b3490814da8170afd7e7e7b599161c965aea7eaa"}
Jul 28 20:08:51.074399 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:51.074381    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:51.090422 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:51.090358    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:08:51.397786 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:51.397539    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wqnw5"]
Jul 28 20:08:51.397954 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:51.397907    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:51.398046 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:51.398024    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:51.401520 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:51.401485    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j2kp2"]
Jul 28 20:08:51.401678 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:51.401612    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:51.401744 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:51.401721    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:51.402757 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:51.402729    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6kmfb"]
Jul 28 20:08:51.402876 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:51.402851    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:51.402974 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:51.402952    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:52.830091 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:52.830055    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:52.830091 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:52.830096    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:52.830740 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:52.830200    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:52.830740 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:52.830309    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:53.829730 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:53.829643    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:53.829934 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:53.829752    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j2kp2" podUID="d953ada0-b545-41e5-901e-4b69b4d33bae"
Jul 28 20:08:54.402185 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:54.402151    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:54.487743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:54.487704    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:54.494759 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:54.494730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:54.494913 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:54.494893    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:54.494967 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:54.494954    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs podName:3ed50e77-948a-4cbd-ae6e-920b8c190c29 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:26.494940158 +0000 UTC m=+66.109177270 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs") pod "network-metrics-daemon-6kmfb" (UID: "3ed50e77-948a-4cbd-ae6e-920b8c190c29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:08:54.595916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:54.595880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshw9\" (UniqueName: \"kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9\") pod \"network-check-target-wqnw5\" (UID: \"503db72d-49cf-4f24-9b5a-3b06f54fbbfc\") " pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:54.596092 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:54.595940    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:54.596092 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:54.596042    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:54.596092 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:54.596056    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:08:54.596092 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:54.596074    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:08:54.596092 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:54.596085    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rshw9 for pod openshift-network-diagnostics/network-check-target-wqnw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:54.596292 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:54.596104    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret podName:d953ada0-b545-41e5-901e-4b69b4d33bae nodeName:}" failed. No retries permitted until 2026-07-28 20:09:26.596086532 +0000 UTC m=+66.210323653 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret") pod "global-pull-secret-syncer-j2kp2" (UID: "d953ada0-b545-41e5-901e-4b69b4d33bae") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:08:54.596292 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:54.596120    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9 podName:503db72d-49cf-4f24-9b5a-3b06f54fbbfc nodeName:}" failed. No retries permitted until 2026-07-28 20:09:26.59611026 +0000 UTC m=+66.210347383 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshw9" (UniqueName: "kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9") pod "network-check-target-wqnw5" (UID: "503db72d-49cf-4f24-9b5a-3b06f54fbbfc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:08:54.830066 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:54.829981    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:54.830258 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:54.830099    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqnw5" podUID="503db72d-49cf-4f24-9b5a-3b06f54fbbfc"
Jul 28 20:08:54.830258 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:54.830155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:54.830384 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:54.830275    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kmfb" podUID="3ed50e77-948a-4cbd-ae6e-920b8c190c29"
Jul 28 20:08:55.690202 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.690177    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-243.ec2.internal" event="NodeReady"
Jul 28 20:08:55.690661 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.690314    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 20:08:55.723807 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.723772    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66788df4d7-prqht"]
Jul 28 20:08:55.745644 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.745621    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ntjln"]
Jul 28 20:08:55.745801 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.745749    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:55.748069 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.748047    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 20:08:55.748186 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.748048    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 20:08:55.748364 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.748346    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cmltp\""
Jul 28 20:08:55.748475 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.748459    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 20:08:55.761602 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.761580    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-spn79"]
Jul 28 20:08:55.761750 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.761731    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:55.762128 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.762105    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 20:08:55.764271 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.764192    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 20:08:55.764271 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.764219    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-488rx\""
Jul 28 20:08:55.764808 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.764220    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 20:08:55.782121 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.782097    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66788df4d7-prqht"]
Jul 28 20:08:55.782262 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.782130    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-spn79"]
Jul 28 20:08:55.782262 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.782146    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ntjln"]
Jul 28 20:08:55.782262 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.782207    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-spn79"
Jul 28 20:08:55.784352 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.784330    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 20:08:55.784472 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.784391    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 20:08:55.784537 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.784512    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k95ng\""
Jul 28 20:08:55.784623 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.784608    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 20:08:55.830243 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.830208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:08:55.833513 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.832710    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 20:08:55.909245 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909199    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6jh9\" (UniqueName: \"kubernetes.io/projected/103df0b3-09df-4d38-8d49-052a14d5556a-kube-api-access-k6jh9\") pod \"ingress-canary-spn79\" (UID: \"103df0b3-09df-4d38-8d49-052a14d5556a\") " pod="openshift-ingress-canary/ingress-canary-spn79"
Jul 28 20:08:55.909430 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909256    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48abf100-5828-4ae4-bb41-d90680574b26-config-volume\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:55.909430 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909309    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2p2\" (UniqueName: \"kubernetes.io/projected/48abf100-5828-4ae4-bb41-d90680574b26-kube-api-access-tk2p2\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:55.909430 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:55.909430 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909414    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/103df0b3-09df-4d38-8d49-052a14d5556a-cert\") pod \"ingress-canary-spn79\" (UID: \"103df0b3-09df-4d38-8d49-052a14d5556a\") " pod="openshift-ingress-canary/ingress-canary-spn79"
Jul 28 20:08:55.909633 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:55.909633 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd821602-a744-4e33-8e4b-5f39739c11e0-ca-trust-extracted\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:55.909633 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909521    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-certificates\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:55.909633 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909557    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-trusted-ca\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:55.909633 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909590    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-installation-pull-secrets\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:55.909633 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghl7w\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-kube-api-access-ghl7w\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:55.909893 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-image-registry-private-configuration\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:55.909893 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909679    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-bound-sa-token\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:55.909893 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:55.909716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48abf100-5828-4ae4-bb41-d90680574b26-tmp-dir\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:56.010444 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-installation-pull-secrets\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.010558 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghl7w\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-kube-api-access-ghl7w\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.010558 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-image-registry-private-configuration\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.010558 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-bound-sa-token\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.010689 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48abf100-5828-4ae4-bb41-d90680574b26-tmp-dir\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:56.010689 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010625    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6jh9\" (UniqueName: \"kubernetes.io/projected/103df0b3-09df-4d38-8d49-052a14d5556a-kube-api-access-k6jh9\") pod \"ingress-canary-spn79\" (UID: \"103df0b3-09df-4d38-8d49-052a14d5556a\") " pod="openshift-ingress-canary/ingress-canary-spn79"
Jul 28 20:08:56.010689 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48abf100-5828-4ae4-bb41-d90680574b26-config-volume\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:56.010873 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2p2\" (UniqueName: \"kubernetes.io/projected/48abf100-5828-4ae4-bb41-d90680574b26-kube-api-access-tk2p2\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:56.010949 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010881    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.010949 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010931    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48abf100-5828-4ae4-bb41-d90680574b26-tmp-dir\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:56.011047 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.010984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/103df0b3-09df-4d38-8d49-052a14d5556a-cert\") pod \"ingress-canary-spn79\" (UID: \"103df0b3-09df-4d38-8d49-052a14d5556a\") " pod="openshift-ingress-canary/ingress-canary-spn79"
Jul 28 20:08:56.011047 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.011010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:56.011047 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:56.011027    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:08:56.011047 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.011039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd821602-a744-4e33-8e4b-5f39739c11e0-ca-trust-extracted\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.011047 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:56.011048    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66788df4d7-prqht: secret "image-registry-tls" not found
Jul 28 20:08:56.011749 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.011073    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-certificates\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.011749 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.011101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-trusted-ca\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.011749 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:56.011297    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls podName:fd821602-a744-4e33-8e4b-5f39739c11e0 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:56.511275975 +0000 UTC m=+36.125513097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls") pod "image-registry-66788df4d7-prqht" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0") : secret "image-registry-tls" not found
Jul 28 20:08:56.011749 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:56.011379    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:08:56.011749 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:56.011426    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls podName:48abf100-5828-4ae4-bb41-d90680574b26 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:56.511410692 +0000 UTC m=+36.125647804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls") pod "dns-default-ntjln" (UID: "48abf100-5828-4ae4-bb41-d90680574b26") : secret "dns-default-metrics-tls" not found
Jul 28 20:08:56.011749 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.011442    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48abf100-5828-4ae4-bb41-d90680574b26-config-volume\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:56.011749 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.011510    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd821602-a744-4e33-8e4b-5f39739c11e0-ca-trust-extracted\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.012149 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.011835    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-certificates\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.012693 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.012675    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-trusted-ca\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.015483 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.015442    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-image-registry-private-configuration\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.015568 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.015479    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-installation-pull-secrets\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.015568 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.015512    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/103df0b3-09df-4d38-8d49-052a14d5556a-cert\") pod \"ingress-canary-spn79\" (UID: \"103df0b3-09df-4d38-8d49-052a14d5556a\") " pod="openshift-ingress-canary/ingress-canary-spn79"
Jul 28 20:08:56.019324 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.019304    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2p2\" (UniqueName: \"kubernetes.io/projected/48abf100-5828-4ae4-bb41-d90680574b26-kube-api-access-tk2p2\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:56.019817 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.019791    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-bound-sa-token\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.020200 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.020065    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghl7w\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-kube-api-access-ghl7w\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.020289 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.020275    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6jh9\" (UniqueName: \"kubernetes.io/projected/103df0b3-09df-4d38-8d49-052a14d5556a-kube-api-access-k6jh9\") pod \"ingress-canary-spn79\" (UID: \"103df0b3-09df-4d38-8d49-052a14d5556a\") " pod="openshift-ingress-canary/ingress-canary-spn79"
Jul 28 20:08:56.090904 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.090866    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac142701-e24f-4e3f-90d4-0367bbafa10c" containerID="889dae8c69010326f46c345fa1eabd5a63608b0649497e6bfe1ebc06cf88c600" exitCode=0
Jul 28 20:08:56.091061 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.090925    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjdxd" event={"ID":"ac142701-e24f-4e3f-90d4-0367bbafa10c","Type":"ContainerDied","Data":"889dae8c69010326f46c345fa1eabd5a63608b0649497e6bfe1ebc06cf88c600"}
Jul 28 20:08:56.091710 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.091584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-spn79"
Jul 28 20:08:56.282048 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.281860    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-spn79"]
Jul 28 20:08:56.311050 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:08:56.311020    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103df0b3_09df_4d38_8d49_052a14d5556a.slice/crio-c400fcfdd4de62cf9b973c9ca5ec9bd66942319917210e4d65e97e21dcd6ca16 WatchSource:0}: Error finding container c400fcfdd4de62cf9b973c9ca5ec9bd66942319917210e4d65e97e21dcd6ca16: Status 404 returned error can't find the container with id c400fcfdd4de62cf9b973c9ca5ec9bd66942319917210e4d65e97e21dcd6ca16
Jul 28 20:08:56.514157 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.514056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:56.514157 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.514125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:56.514403 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:56.514249    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:08:56.514403 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:56.514306    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls podName:48abf100-5828-4ae4-bb41-d90680574b26 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:57.51429265 +0000 UTC m=+37.128529761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls") pod "dns-default-ntjln" (UID: "48abf100-5828-4ae4-bb41-d90680574b26") : secret "dns-default-metrics-tls" not found
Jul 28 20:08:56.514403 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:56.514248    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:08:56.514403 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:56.514346    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66788df4d7-prqht: secret "image-registry-tls" not found
Jul 28 20:08:56.514539 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:56.514407    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls podName:fd821602-a744-4e33-8e4b-5f39739c11e0 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:57.514387283 +0000 UTC m=+37.128624410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls") pod "image-registry-66788df4d7-prqht" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0") : secret "image-registry-tls" not found
Jul 28 20:08:56.829411 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.829341    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:08:56.829411 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.829378    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:08:56.831580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.831557    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 20:08:56.831580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.831568    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 20:08:56.831765 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.831693    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 20:08:56.831765 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.831706    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82krx\""
Jul 28 20:08:56.831864 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:56.831814    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sp59p\""
Jul 28 20:08:57.097081 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:57.097002    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac142701-e24f-4e3f-90d4-0367bbafa10c" containerID="dda5285b3ab9fe9e341c70c88412ed3a601c31e3687e0067cf50b62f1ebb8560" exitCode=0
Jul 28 20:08:57.097081 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:57.097050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjdxd" event={"ID":"ac142701-e24f-4e3f-90d4-0367bbafa10c","Type":"ContainerDied","Data":"dda5285b3ab9fe9e341c70c88412ed3a601c31e3687e0067cf50b62f1ebb8560"}
Jul 28 20:08:57.098408 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:57.098386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-spn79" event={"ID":"103df0b3-09df-4d38-8d49-052a14d5556a","Type":"ContainerStarted","Data":"c400fcfdd4de62cf9b973c9ca5ec9bd66942319917210e4d65e97e21dcd6ca16"}
Jul 28 20:08:57.522160 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:57.522119    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:57.522361 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:57.522214    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:57.522361 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:57.522284    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:08:57.522361 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:57.522308    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66788df4d7-prqht: secret "image-registry-tls" not found
Jul 28 20:08:57.522361 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:57.522359    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:08:57.522580 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:57.522371    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls podName:fd821602-a744-4e33-8e4b-5f39739c11e0 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:59.522352414 +0000 UTC m=+39.136589541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls") pod "image-registry-66788df4d7-prqht" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0") : secret "image-registry-tls" not found
Jul 28 20:08:57.522580 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:57.522404    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls podName:48abf100-5828-4ae4-bb41-d90680574b26 nodeName:}" failed. No retries permitted until 2026-07-28 20:08:59.52239324 +0000 UTC m=+39.136630352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls") pod "dns-default-ntjln" (UID: "48abf100-5828-4ae4-bb41-d90680574b26") : secret "dns-default-metrics-tls" not found
Jul 28 20:08:57.819751 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:57.819664    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:57.883821 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:57.883788    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:58.105067 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:58.104985    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjdxd" event={"ID":"ac142701-e24f-4e3f-90d4-0367bbafa10c","Type":"ContainerStarted","Data":"c2354715b6f7eb8bc6548651eafcf8e6b1dfb67087e4be07aa68b1b470cfd43d"}
Jul 28 20:08:58.131523 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:58.131458    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tjdxd" podStartSLOduration=7.653341559 podStartE2EDuration="37.131435699s" totalImagesPullingTime="29.47809414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:58.130710585 +0000 UTC m=+37.744947720" watchObservedRunningTime="2026-07-28 20:08:58.131435699 +0000 UTC m=+37.745672834"
Jul 28 20:08:58.262163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:58.261927    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:58.405860 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:58.405823    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:59.108554 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:59.108516    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-spn79" event={"ID":"103df0b3-09df-4d38-8d49-052a14d5556a","Type":"ContainerStarted","Data":"f6f3a8c64f0caa95a44b142e02c00620814b7129d9bf3567c9ffa35a434468ef"}
Jul 28 20:08:59.124601 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:59.124559    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-spn79" podStartSLOduration=2.016731445 podStartE2EDuration="4.124543181s" totalImagesPullingTime="2.107811736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:59.124071295 +0000 UTC m=+38.738308433" watchObservedRunningTime="2026-07-28 20:08:59.124543181 +0000 UTC m=+38.738780314"
Jul 28 20:08:59.537638 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:59.537598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:08:59.537818 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:59.537671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:08:59.537818 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:59.537741    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:08:59.537818 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:59.537762    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66788df4d7-prqht: secret "image-registry-tls" not found
Jul 28 20:08:59.537818 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:59.537806    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:08:59.537818 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:59.537817    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls podName:fd821602-a744-4e33-8e4b-5f39739c11e0 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:03.537801264 +0000 UTC m=+43.152038402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls") pod "image-registry-66788df4d7-prqht" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0") : secret "image-registry-tls" not found
Jul 28 20:08:59.537995 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:08:59.537863    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls podName:48abf100-5828-4ae4-bb41-d90680574b26 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:03.537850071 +0000 UTC m=+43.152087182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls") pod "dns-default-ntjln" (UID: "48abf100-5828-4ae4-bb41-d90680574b26") : secret "dns-default-metrics-tls" not found
Jul 28 20:08:59.786172 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:59.786136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:59.841366 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:08:59.841278    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:01.431008 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.430971    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:01.484761 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.484730    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:01.719538 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.719454    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:01.758580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.758547    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p"]
Jul 28 20:09:01.764217 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.764201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p"
Jul 28 20:09:01.766522 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.766500    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-s52fb\""
Jul 28 20:09:01.766522 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.766500    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 20:09:01.767442 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.767411    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 20:09:01.773400 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.773370    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p"]
Jul 28 20:09:01.810991 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.810961    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rgnhx"]
Jul 28 20:09:01.815439 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.815420    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:01.815929 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.815892    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:01.817524 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.817505    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 20:09:01.817602 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.817505    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 20:09:01.817602 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.817543    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 20:09:01.817602 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.817556    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tn5xs\""
Jul 28 20:09:01.817758 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.817603    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 20:09:01.823078 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.823057    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rgnhx"]
Jul 28 20:09:01.856295 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.856251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvgf\" (UniqueName: \"kubernetes.io/projected/4d05786a-a870-4963-931a-9bfd0abebb1c-kube-api-access-2pvgf\") pod \"migrator-6865c9967-6db8p\" (UID: \"4d05786a-a870-4963-931a-9bfd0abebb1c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p"
Jul 28 20:09:01.925266 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.925211    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:01.956832 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.956799    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-crio-socket\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:01.957001 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.956869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:01.957001 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.956919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvgf\" (UniqueName: \"kubernetes.io/projected/4d05786a-a870-4963-931a-9bfd0abebb1c-kube-api-access-2pvgf\") pod \"migrator-6865c9967-6db8p\" (UID: \"4d05786a-a870-4963-931a-9bfd0abebb1c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p"
Jul 28 20:09:01.957001 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.956949    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:01.957001 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.956988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksf5k\" (UniqueName: \"kubernetes.io/projected/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-api-access-ksf5k\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:01.957152 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.957059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-data-volume\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:01.975124 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:01.975047    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvgf\" (UniqueName: \"kubernetes.io/projected/4d05786a-a870-4963-931a-9bfd0abebb1c-kube-api-access-2pvgf\") pod \"migrator-6865c9967-6db8p\" (UID: \"4d05786a-a870-4963-931a-9bfd0abebb1c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p"
Jul 28 20:09:02.058312 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.058269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-crio-socket\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:02.058512 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.058342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:02.058512 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:02.058429    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:09:02.058512 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:02.058480    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls podName:898ae9a8-22e1-41de-90c4-ad47d2ac4e47 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:02.558466794 +0000 UTC m=+42.172703905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgnhx" (UID: "898ae9a8-22e1-41de-90c4-ad47d2ac4e47") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:09:02.058512 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.058492    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:02.058512 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.058514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksf5k\" (UniqueName: \"kubernetes.io/projected/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-api-access-ksf5k\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:02.058755 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.058532    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-data-volume\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:02.058755 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.058541    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-crio-socket\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:02.058755 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:02.058604    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:09:02.058755 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:02.058672    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm podName:898ae9a8-22e1-41de-90c4-ad47d2ac4e47 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:02.558660119 +0000 UTC m=+42.172897234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgnhx" (UID: "898ae9a8-22e1-41de-90c4-ad47d2ac4e47") : configmap "kube-rbac-proxy" not found
Jul 28 20:09:02.058946 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.058895    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-data-volume\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:02.069266 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.069240    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksf5k\" (UniqueName: \"kubernetes.io/projected/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-api-access-ksf5k\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:02.075301 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.075282    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p"
Jul 28 20:09:02.155854 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.155777    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:02.196422 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.196391    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p"]
Jul 28 20:09:02.199541 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:09:02.199513    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d05786a_a870_4963_931a_9bfd0abebb1c.slice/crio-1b9133dee2ae3ea155c7b41d7b00619d2fa34fc22b708254659d28f504df7541 WatchSource:0}: Error finding container 1b9133dee2ae3ea155c7b41d7b00619d2fa34fc22b708254659d28f504df7541: Status 404 returned error can't find the container with id 1b9133dee2ae3ea155c7b41d7b00619d2fa34fc22b708254659d28f504df7541
Jul 28 20:09:02.258067 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.257976    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:02.563384 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.563300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:02.563743 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:02.563429    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:09:02.563743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.563470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:02.563743 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:02.563491    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm podName:898ae9a8-22e1-41de-90c4-ad47d2ac4e47 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:03.563476551 +0000 UTC m=+43.177713662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgnhx" (UID: "898ae9a8-22e1-41de-90c4-ad47d2ac4e47") : configmap "kube-rbac-proxy" not found
Jul 28 20:09:02.563743 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:02.563557    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:09:02.563743 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:02.563611    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls podName:898ae9a8-22e1-41de-90c4-ad47d2ac4e47 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:03.563596779 +0000 UTC m=+43.177833893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgnhx" (UID: "898ae9a8-22e1-41de-90c4-ad47d2ac4e47") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:09:02.597869 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.597822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:02.666033 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.666002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:02.923314 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:02.923278    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:03.023027 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:03.022984    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:03.120868 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:03.120832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p" event={"ID":"4d05786a-a870-4963-931a-9bfd0abebb1c","Type":"ContainerStarted","Data":"1b9133dee2ae3ea155c7b41d7b00619d2fa34fc22b708254659d28f504df7541"}
Jul 28 20:09:03.572416 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:03.572373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:03.572436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:03.572485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:03.572512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:03.572543    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:03.572567    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66788df4d7-prqht: secret "image-registry-tls" not found
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:03.572594    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:03.572611    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:03.572625    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls podName:fd821602-a744-4e33-8e4b-5f39739c11e0 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:11.572604644 +0000 UTC m=+51.186841759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls") pod "image-registry-66788df4d7-prqht" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0") : secret "image-registry-tls" not found
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:03.572648    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm podName:898ae9a8-22e1-41de-90c4-ad47d2ac4e47 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:05.572631359 +0000 UTC m=+45.186868475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgnhx" (UID: "898ae9a8-22e1-41de-90c4-ad47d2ac4e47") : configmap "kube-rbac-proxy" not found
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:03.572611    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:03.572668    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls podName:898ae9a8-22e1-41de-90c4-ad47d2ac4e47 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:05.572658749 +0000 UTC m=+45.186895864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgnhx" (UID: "898ae9a8-22e1-41de-90c4-ad47d2ac4e47") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:09:03.572896 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:03.572689    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls podName:48abf100-5828-4ae4-bb41-d90680574b26 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:11.572673234 +0000 UTC m=+51.186910358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls") pod "dns-default-ntjln" (UID: "48abf100-5828-4ae4-bb41-d90680574b26") : secret "dns-default-metrics-tls" not found
Jul 28 20:09:04.124372 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.124347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p" event={"ID":"4d05786a-a870-4963-931a-9bfd0abebb1c","Type":"ContainerStarted","Data":"e455c38cb090aa0dcef35341b71aaecbf576746928126bf02a2e7b2cfae7f611"}
Jul 28 20:09:04.418270 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.418218    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n87ww_f52174fd-3b1a-465b-b0f4-4068deab6013/dns-node-resolver/0.log"
Jul 28 20:09:04.461452 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.461422    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mdfgn"]
Jul 28 20:09:04.484959 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.484935    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mdfgn"]
Jul 28 20:09:04.485086 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.485026    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.487413 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.487388    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 20:09:04.487413 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.487395    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dqxls\""
Jul 28 20:09:04.488180 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.488163    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 20:09:04.488264 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.488184    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 20:09:04.488264 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.488164    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 20:09:04.496343 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.496312    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:04.560743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.560703    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:04.580932 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.580896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8t8w\" (UniqueName: \"kubernetes.io/projected/ec8b8809-355f-406d-b178-f94a03505700-kube-api-access-g8t8w\") pod \"service-ca-658d7b7c78-mdfgn\" (UID: \"ec8b8809-355f-406d-b178-f94a03505700\") " pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.581259 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.580948    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ec8b8809-355f-406d-b178-f94a03505700-signing-key\") pod \"service-ca-658d7b7c78-mdfgn\" (UID: \"ec8b8809-355f-406d-b178-f94a03505700\") " pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.581259 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.581011    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ec8b8809-355f-406d-b178-f94a03505700-signing-cabundle\") pod \"service-ca-658d7b7c78-mdfgn\" (UID: \"ec8b8809-355f-406d-b178-f94a03505700\") " pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.682044 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.681958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8t8w\" (UniqueName: \"kubernetes.io/projected/ec8b8809-355f-406d-b178-f94a03505700-kube-api-access-g8t8w\") pod \"service-ca-658d7b7c78-mdfgn\" (UID: \"ec8b8809-355f-406d-b178-f94a03505700\") " pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.682044 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.681999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ec8b8809-355f-406d-b178-f94a03505700-signing-key\") pod \"service-ca-658d7b7c78-mdfgn\" (UID: \"ec8b8809-355f-406d-b178-f94a03505700\") " pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.682044 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.682031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ec8b8809-355f-406d-b178-f94a03505700-signing-cabundle\") pod \"service-ca-658d7b7c78-mdfgn\" (UID: \"ec8b8809-355f-406d-b178-f94a03505700\") " pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.682752 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.682733    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ec8b8809-355f-406d-b178-f94a03505700-signing-cabundle\") pod \"service-ca-658d7b7c78-mdfgn\" (UID: \"ec8b8809-355f-406d-b178-f94a03505700\") " pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.690249 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.690207    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8t8w\" (UniqueName: \"kubernetes.io/projected/ec8b8809-355f-406d-b178-f94a03505700-kube-api-access-g8t8w\") pod \"service-ca-658d7b7c78-mdfgn\" (UID: \"ec8b8809-355f-406d-b178-f94a03505700\") " pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.698460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.698436    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ec8b8809-355f-406d-b178-f94a03505700-signing-key\") pod \"service-ca-658d7b7c78-mdfgn\" (UID: \"ec8b8809-355f-406d-b178-f94a03505700\") " pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.796576 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.796538    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn"
Jul 28 20:09:04.924725 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:04.924693    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mdfgn"]
Jul 28 20:09:04.928370 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:09:04.928346    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8b8809_355f_406d_b178_f94a03505700.slice/crio-f033cc6df61b1f88b6f7173956cf52a78cf377df244a436c39021a4a0533bf9c WatchSource:0}: Error finding container f033cc6df61b1f88b6f7173956cf52a78cf377df244a436c39021a4a0533bf9c: Status 404 returned error can't find the container with id f033cc6df61b1f88b6f7173956cf52a78cf377df244a436c39021a4a0533bf9c
Jul 28 20:09:05.021120 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:05.021048    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4db9q_88a7f2ff-c2cc-4b29-8bf2-66861c7931bc/node-ca/0.log"
Jul 28 20:09:05.128953 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:05.128906    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p" event={"ID":"4d05786a-a870-4963-931a-9bfd0abebb1c","Type":"ContainerStarted","Data":"49a08d54834440d9dd9dbe196dac79d00ee2e65e77a246734675103c12e812d9"}
Jul 28 20:09:05.129965 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:05.129933    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn" event={"ID":"ec8b8809-355f-406d-b178-f94a03505700","Type":"ContainerStarted","Data":"f033cc6df61b1f88b6f7173956cf52a78cf377df244a436c39021a4a0533bf9c"}
Jul 28 20:09:05.148867 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:05.148820    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6db8p" podStartSLOduration=2.360403483 podStartE2EDuration="4.148804619s" totalImagesPullingTime="1.788401136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:05.147411396 +0000 UTC m=+44.761648540" watchObservedRunningTime="2026-07-28 20:09:05.148804619 +0000 UTC m=+44.763041751"
Jul 28 20:09:05.473080 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:05.473046    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:05.590555 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:05.590516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:05.591003 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:05.590592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:05.591003 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:05.590688    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:09:05.591003 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:05.590758    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm podName:898ae9a8-22e1-41de-90c4-ad47d2ac4e47 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:09.590739432 +0000 UTC m=+49.204976558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgnhx" (UID: "898ae9a8-22e1-41de-90c4-ad47d2ac4e47") : configmap "kube-rbac-proxy" not found
Jul 28 20:09:05.591003 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:05.590687    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:09:05.591003 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:05.590839    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls podName:898ae9a8-22e1-41de-90c4-ad47d2ac4e47 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:09.590826038 +0000 UTC m=+49.205063153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgnhx" (UID: "898ae9a8-22e1-41de-90c4-ad47d2ac4e47") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:09:06.020383 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:06.020352    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-spn79_103df0b3-09df-4d38-8d49-052a14d5556a/serve-healthcheck-canary/0.log"
Jul 28 20:09:06.218829 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:06.218798    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-6db8p_4d05786a-a870-4963-931a-9bfd0abebb1c/migrator/0.log"
Jul 28 20:09:06.418761 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:06.418736    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-6db8p_4d05786a-a870-4963-931a-9bfd0abebb1c/graceful-termination/0.log"
Jul 28 20:09:07.031585 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:07.031548    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:07.093894 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:07.093865    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:08.138369 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:08.138338    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn" event={"ID":"ec8b8809-355f-406d-b178-f94a03505700","Type":"ContainerStarted","Data":"5c37b9c5446c812d2de586ee979d039fcc0b0ca192d4704cb27c72e3556dcdfe"}
Jul 28 20:09:08.157391 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:08.157346    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-mdfgn" podStartSLOduration=1.76945712 podStartE2EDuration="4.157330589s" totalImagesPullingTime="2.387873469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:08.156202613 +0000 UTC m=+47.770439747" watchObservedRunningTime="2026-07-28 20:09:08.157330589 +0000 UTC m=+47.771567756"
Jul 28 20:09:08.795496 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:08.795461    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:08.896940 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:08.896896    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:09.273284 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:09.273246    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:09.411311 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:09.411277    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:09.624057 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:09.623966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:09.624057 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:09.624048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:09.624721 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:09.624686    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:09.625264 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:09.625217    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:09:09.625382 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:09.625366    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls podName:898ae9a8-22e1-41de-90c4-ad47d2ac4e47 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:17.625339639 +0000 UTC m=+57.239576766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgnhx" (UID: "898ae9a8-22e1-41de-90c4-ad47d2ac4e47") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:09:09.636407 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:09.636375    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:09.734526 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:09.734491    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:09.849184 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:09.849152    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:09.935182 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:09.935150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:09.989510 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:09.989477    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:10.559238 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:10.559193    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:11.161162 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:11.161097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:11.572544 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:11.572441    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:11.642192 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:11.642156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:09:11.642406 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:11.642249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:09:11.642406 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:11.642334    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:09:11.642406 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:11.642353    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:09:11.642406 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:11.642365    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66788df4d7-prqht: secret "image-registry-tls" not found
Jul 28 20:09:11.642406 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:11.642410    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls podName:fd821602-a744-4e33-8e4b-5f39739c11e0 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:27.642393688 +0000 UTC m=+67.256630803 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls") pod "image-registry-66788df4d7-prqht" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0") : secret "image-registry-tls" not found
Jul 28 20:09:11.642681 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:11.642424    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls podName:48abf100-5828-4ae4-bb41-d90680574b26 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:27.642417244 +0000 UTC m=+67.256654356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls") pod "dns-default-ntjln" (UID: "48abf100-5828-4ae4-bb41-d90680574b26") : secret "dns-default-metrics-tls" not found
Jul 28 20:09:11.966009 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:11.965973    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:12.157659 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:12.157619    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:12.763672 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:12.763645    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:12.962105 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:12.962071    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:13.221084 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:13.221046    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:13.362160 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:13.362126    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:13.560279 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:13.560173    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:13.761901 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:13.761865    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:13.823934 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:13.823861    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:14.164381 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:14.164349    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:14.562175 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:14.562090    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:14.663917 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:14.663881    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:14.717851 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:14.717823    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:15.362409 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:15.362371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:15.466988 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:15.466952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:15.711281 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:15.711247    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:15.814017 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:15.813981    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:15.877362 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:15.877328    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:15.962641 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:15.962559    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:16.063241 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:16.063208    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:16.187616 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:16.187591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:16.244293 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:16.244213    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:16.383555 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:16.383525    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:16.560689 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:16.560617    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:16.677366 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:16.677336    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:16.802238 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:16.802208    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:16.905735 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:16.905706    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:17.008998 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:17.008964    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:17.104172 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:17.104142    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:17.396459 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:17.396430    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:17.592649 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:17.592619    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:17.690322 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:17.690221    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:17.692513 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:17.692484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/898ae9a8-22e1-41de-90c4-ad47d2ac4e47-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgnhx\" (UID: \"898ae9a8-22e1-41de-90c4-ad47d2ac4e47\") " pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:17.724684 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:17.724650    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rgnhx"
Jul 28 20:09:17.763395 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:17.763354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:17.871193 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:17.871165    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rgnhx"]
Jul 28 20:09:17.875143 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:09:17.875115    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898ae9a8_22e1_41de_90c4_ad47d2ac4e47.slice/crio-6efc2e468cf2134de3956699887fcb24bac4ea026a54fba63482a5b862160395 WatchSource:0}: Error finding container 6efc2e468cf2134de3956699887fcb24bac4ea026a54fba63482a5b862160395: Status 404 returned error can't find the container with id 6efc2e468cf2134de3956699887fcb24bac4ea026a54fba63482a5b862160395
Jul 28 20:09:17.963974 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:17.963869    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:18.165015 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:18.164984    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgnhx" event={"ID":"898ae9a8-22e1-41de-90c4-ad47d2ac4e47","Type":"ContainerStarted","Data":"6efc2e468cf2134de3956699887fcb24bac4ea026a54fba63482a5b862160395"}
Jul 28 20:09:18.367967 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:18.367861    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:18.559949 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:18.559918    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:18.810131 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:18.810029    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:18.911211 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:18.911142    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:19.158822 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:19.158786    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:19.563439 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:19.563360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:19.764668 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:19.764630    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:20.203384 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:20.203347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:20.361200 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:20.361154    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:22.027190 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:22.027142    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:22.086786 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:22.086749    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:23.089329 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:23.089305    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9zbl"
Jul 28 20:09:23.181995 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:23.181953    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgnhx" event={"ID":"898ae9a8-22e1-41de-90c4-ad47d2ac4e47","Type":"ContainerStarted","Data":"051eb3b4ec484f90f60c54c1061d3b32fc61c4f13316abbe2bdf4c06287bc7e7"}
Jul 28 20:09:23.210241 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:23.210193    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:23.310896 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:23.310867    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:24.520409 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:24.520368    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:24.579332 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:24.579303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:26.558372 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.558340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:09:26.560965 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.560946    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 20:09:26.571156 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.571129    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed50e77-948a-4cbd-ae6e-920b8c190c29-metrics-certs\") pod \"network-metrics-daemon-6kmfb\" (UID: \"3ed50e77-948a-4cbd-ae6e-920b8c190c29\") " pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:09:26.659404 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.659364    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:09:26.659575 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.659430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshw9\" (UniqueName: \"kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9\") pod \"network-check-target-wqnw5\" (UID: \"503db72d-49cf-4f24-9b5a-3b06f54fbbfc\") " pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:09:26.661593 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.661576    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 20:09:26.661637 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.661619    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 20:09:26.672465 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.672438    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 20:09:26.672664 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.672646    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d953ada0-b545-41e5-901e-4b69b4d33bae-original-pull-secret\") pod \"global-pull-secret-syncer-j2kp2\" (UID: \"d953ada0-b545-41e5-901e-4b69b4d33bae\") " pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:09:26.683074 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.683053    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshw9\" (UniqueName: \"kubernetes.io/projected/503db72d-49cf-4f24-9b5a-3b06f54fbbfc-kube-api-access-rshw9\") pod \"network-check-target-wqnw5\" (UID: \"503db72d-49cf-4f24-9b5a-3b06f54fbbfc\") " pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:09:26.741317 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.741210    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j2kp2"
Jul 28 20:09:26.841646 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.841616    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82krx\""
Jul 28 20:09:26.845287 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.845265    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sp59p\""
Jul 28 20:09:26.850043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.850018    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kmfb"
Jul 28 20:09:26.853852 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.853832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:09:26.856979 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.856954    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:26.870325 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.870298    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j2kp2"]
Jul 28 20:09:26.874531 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:09:26.874496    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd953ada0_b545_41e5_901e_4b69b4d33bae.slice/crio-fda3096b52d095efd83c9313b0c78e66aae31ca0e5c1b012046b19842276c532 WatchSource:0}: Error finding container fda3096b52d095efd83c9313b0c78e66aae31ca0e5c1b012046b19842276c532: Status 404 returned error can't find the container with id fda3096b52d095efd83c9313b0c78e66aae31ca0e5c1b012046b19842276c532
Jul 28 20:09:26.912977 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.910311    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66788df4d7-prqht"]
Jul 28 20:09:26.912977 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:09:26.910617    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-66788df4d7-prqht" podUID="fd821602-a744-4e33-8e4b-5f39739c11e0"
Jul 28 20:09:26.929902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.929644    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"]
Jul 28 20:09:26.933006 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.932866    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"]
Jul 28 20:09:26.933132 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.933024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"
Jul 28 20:09:26.935606 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.935585    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:26.936075 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.936037    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bk2sl\""
Jul 28 20:09:26.936281 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.936261    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 20:09:26.936488 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.936265    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 20:09:26.938296 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.938020    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 20:09:26.938296 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.938084    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 20:09:26.938447 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.938346    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 20:09:26.938743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.938514    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 20:09:26.938743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.938668    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 20:09:26.939119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.938949    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 20:09:26.944994 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.944948    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"]
Jul 28 20:09:26.948305 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.948221    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"]
Jul 28 20:09:26.962462 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.962285    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:26.962462 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.962341    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:26.962462 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.962370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-ca\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:26.962462 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.962397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bd0f15-554c-43eb-8052-4592d55159be-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:26.962462 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.962434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trh8\" (UniqueName: \"kubernetes.io/projected/b4de61c9-a562-41e0-9b23-84439ed65a99-kube-api-access-9trh8\") pod \"managed-serviceaccount-addon-agent-985b44966-qxtzv\" (UID: \"b4de61c9-a562-41e0-9b23-84439ed65a99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"
Jul 28 20:09:26.962758 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.962543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-hub\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:26.962758 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.962603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4de61c9-a562-41e0-9b23-84439ed65a99-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-985b44966-qxtzv\" (UID: \"b4de61c9-a562-41e0-9b23-84439ed65a99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"
Jul 28 20:09:26.962758 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.962653    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4v2\" (UniqueName: \"kubernetes.io/projected/d7bd0f15-554c-43eb-8052-4592d55159be-kube-api-access-9f4v2\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:26.963347 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:26.963328    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:27.020470 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.020389    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6kmfb"]
Jul 28 20:09:27.024416 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:09:27.024388    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed50e77_948a_4cbd_ae6e_920b8c190c29.slice/crio-11a8334a642bacaa220ff6edf2be294399951f7835dd0c5a73ba5a2857ccd2bc WatchSource:0}: Error finding container 11a8334a642bacaa220ff6edf2be294399951f7835dd0c5a73ba5a2857ccd2bc: Status 404 returned error can't find the container with id 11a8334a642bacaa220ff6edf2be294399951f7835dd0c5a73ba5a2857ccd2bc
Jul 28 20:09:27.026930 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.026911    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66674cb6c9-wngfz"]
Jul 28 20:09:27.031698 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.031676    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.047646 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.047622    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66674cb6c9-wngfz"]
Jul 28 20:09:27.054784 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.054760    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wqnw5"]
Jul 28 20:09:27.059030 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:09:27.059008    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503db72d_49cf_4f24_9b5a_3b06f54fbbfc.slice/crio-fc83b20ab61d42a5937f65cfb2cf93709c5487f09aec70962d01d48771f14d3d WatchSource:0}: Error finding container fc83b20ab61d42a5937f65cfb2cf93709c5487f09aec70962d01d48771f14d3d: Status 404 returned error can't find the container with id fc83b20ab61d42a5937f65cfb2cf93709c5487f09aec70962d01d48771f14d3d
Jul 28 20:09:27.063589 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063553    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.063679 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-image-registry-private-configuration\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.063679 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063634    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-ca\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.063679 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bd0f15-554c-43eb-8052-4592d55159be-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.063856 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-installation-pull-secrets\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.063856 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063775    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trh8\" (UniqueName: \"kubernetes.io/projected/b4de61c9-a562-41e0-9b23-84439ed65a99-kube-api-access-9trh8\") pod \"managed-serviceaccount-addon-agent-985b44966-qxtzv\" (UID: \"b4de61c9-a562-41e0-9b23-84439ed65a99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"
Jul 28 20:09:27.063856 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-ca-trust-extracted\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.064013 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-hub\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.064013 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4de61c9-a562-41e0-9b23-84439ed65a99-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-985b44966-qxtzv\" (UID: \"b4de61c9-a562-41e0-9b23-84439ed65a99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"
Jul 28 20:09:27.064013 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4v2\" (UniqueName: \"kubernetes.io/projected/d7bd0f15-554c-43eb-8052-4592d55159be-kube-api-access-9f4v2\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.064013 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.063997    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-trusted-ca\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.064212 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.064026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-bound-sa-token\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.064212 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.064071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-registry-tls\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.064212 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.064096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp49z\" (UniqueName: \"kubernetes.io/projected/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-kube-api-access-kp49z\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.064212 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.064124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-registry-certificates\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.064212 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.064156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.065168 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.065125    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bd0f15-554c-43eb-8052-4592d55159be-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.066556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.066517    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-ca\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.066799 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.066780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.066877 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.066797    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.067039 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.067016    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d7bd0f15-554c-43eb-8052-4592d55159be-hub\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.067247 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.067214    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4de61c9-a562-41e0-9b23-84439ed65a99-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-985b44966-qxtzv\" (UID: \"b4de61c9-a562-41e0-9b23-84439ed65a99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"
Jul 28 20:09:27.069952 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.069928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:27.074368 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.074346    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4v2\" (UniqueName: \"kubernetes.io/projected/d7bd0f15-554c-43eb-8052-4592d55159be-kube-api-access-9f4v2\") pod \"cluster-proxy-proxy-agent-7c6fc98875-rgdls\" (UID: \"d7bd0f15-554c-43eb-8052-4592d55159be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.074752 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.074729    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trh8\" (UniqueName: \"kubernetes.io/projected/b4de61c9-a562-41e0-9b23-84439ed65a99-kube-api-access-9trh8\") pod \"managed-serviceaccount-addon-agent-985b44966-qxtzv\" (UID: \"b4de61c9-a562-41e0-9b23-84439ed65a99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"
Jul 28 20:09:27.158366 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.158327    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:27.165587 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.165564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-installation-pull-secrets\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.165691 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.165599    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-ca-trust-extracted\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.165691 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.165650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-trusted-ca\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.165691 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.165674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-bound-sa-token\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.165854 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.165710    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-registry-tls\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.165854 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.165732    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp49z\" (UniqueName: \"kubernetes.io/projected/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-kube-api-access-kp49z\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.165854 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.165757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-registry-certificates\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.165854 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.165803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-image-registry-private-configuration\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.166117 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.166090    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-ca-trust-extracted\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.167009 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.166961    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-registry-certificates\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.167128 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.167085    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-trusted-ca\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.168324 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.168291    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-installation-pull-secrets\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.168544 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.168521    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-registry-tls\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.168617 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.168565    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-image-registry-private-configuration\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.177965 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.177941    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-bound-sa-token\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.178163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.178144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp49z\" (UniqueName: \"kubernetes.io/projected/0864b1c1-412c-4667-9cbd-58ae7d7f0de3-kube-api-access-kp49z\") pod \"image-registry-66674cb6c9-wngfz\" (UID: \"0864b1c1-412c-4667-9cbd-58ae7d7f0de3\") " pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.195756 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.195725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgnhx" event={"ID":"898ae9a8-22e1-41de-90c4-ad47d2ac4e47","Type":"ContainerStarted","Data":"c889fe1d2bf046f8e466a806d8f0f5854cf6b6ab3997defeab687ae0e1600e1b"}
Jul 28 20:09:27.197127 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.197072    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kmfb" event={"ID":"3ed50e77-948a-4cbd-ae6e-920b8c190c29","Type":"ContainerStarted","Data":"11a8334a642bacaa220ff6edf2be294399951f7835dd0c5a73ba5a2857ccd2bc"}
Jul 28 20:09:27.198283 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.198252    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wqnw5" event={"ID":"503db72d-49cf-4f24-9b5a-3b06f54fbbfc","Type":"ContainerStarted","Data":"fc83b20ab61d42a5937f65cfb2cf93709c5487f09aec70962d01d48771f14d3d"}
Jul 28 20:09:27.199385 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.199363    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j2kp2" event={"ID":"d953ada0-b545-41e5-901e-4b69b4d33bae","Type":"ContainerStarted","Data":"fda3096b52d095efd83c9313b0c78e66aae31ca0e5c1b012046b19842276c532"}
Jul 28 20:09:27.199495 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.199399    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:09:27.206261 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.206216    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:09:27.259152 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.259101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:27.266054 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266027    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-certificates\") pod \"fd821602-a744-4e33-8e4b-5f39739c11e0\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") "
Jul 28 20:09:27.266197 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266101    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-image-registry-private-configuration\") pod \"fd821602-a744-4e33-8e4b-5f39739c11e0\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") "
Jul 28 20:09:27.266197 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266140    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-installation-pull-secrets\") pod \"fd821602-a744-4e33-8e4b-5f39739c11e0\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") "
Jul 28 20:09:27.266197 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266180    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd821602-a744-4e33-8e4b-5f39739c11e0-ca-trust-extracted\") pod \"fd821602-a744-4e33-8e4b-5f39739c11e0\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") "
Jul 28 20:09:27.266386 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266216    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghl7w\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-kube-api-access-ghl7w\") pod \"fd821602-a744-4e33-8e4b-5f39739c11e0\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") "
Jul 28 20:09:27.266386 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266255    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-bound-sa-token\") pod \"fd821602-a744-4e33-8e4b-5f39739c11e0\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") "
Jul 28 20:09:27.266386 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266309    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-trusted-ca\") pod \"fd821602-a744-4e33-8e4b-5f39739c11e0\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") "
Jul 28 20:09:27.266386 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266348    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fd821602-a744-4e33-8e4b-5f39739c11e0" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:09:27.266515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266497    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd821602-a744-4e33-8e4b-5f39739c11e0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fd821602-a744-4e33-8e4b-5f39739c11e0" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:09:27.266598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266550    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-certificates\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:09:27.266949 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.266909    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fd821602-a744-4e33-8e4b-5f39739c11e0" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:09:27.268813 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.268727    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fd821602-a744-4e33-8e4b-5f39739c11e0" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:09:27.268813 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.268737    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-kube-api-access-ghl7w" (OuterVolumeSpecName: "kube-api-access-ghl7w") pod "fd821602-a744-4e33-8e4b-5f39739c11e0" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0"). InnerVolumeSpecName "kube-api-access-ghl7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:09:27.268982 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.268864    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fd821602-a744-4e33-8e4b-5f39739c11e0" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:09:27.268982 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.268874    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fd821602-a744-4e33-8e4b-5f39739c11e0" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:09:27.278143 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.278083    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"
Jul 28 20:09:27.289199 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.289180    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"
Jul 28 20:09:27.358766 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.358590    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cmltp\""
Jul 28 20:09:27.360459 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.360256    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:27.370088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.367667    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-installation-pull-secrets\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:09:27.370088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.367694    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd821602-a744-4e33-8e4b-5f39739c11e0-ca-trust-extracted\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:09:27.370088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.367709    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghl7w\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-kube-api-access-ghl7w\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:09:27.370088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.367721    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-bound-sa-token\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:09:27.370088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.367736    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd821602-a744-4e33-8e4b-5f39739c11e0-trusted-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:09:27.370088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.367749    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd821602-a744-4e33-8e4b-5f39739c11e0-image-registry-private-configuration\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:09:27.370088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.367805    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:27.459871 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.459443    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv"]
Jul 28 20:09:27.476777 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.476422    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls"]
Jul 28 20:09:27.478639 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:09:27.478614    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7bd0f15_554c_43eb_8052_4592d55159be.slice/crio-14ae6c812a0493001a921bcc109e4fcef62396421491e779b40c53fedeef3a15 WatchSource:0}: Error finding container 14ae6c812a0493001a921bcc109e4fcef62396421491e779b40c53fedeef3a15: Status 404 returned error can't find the container with id 14ae6c812a0493001a921bcc109e4fcef62396421491e779b40c53fedeef3a15
Jul 28 20:09:27.546866 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.546562    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66674cb6c9-wngfz"]
Jul 28 20:09:27.553247 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:09:27.553202    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0864b1c1_412c_4667_9cbd_58ae7d7f0de3.slice/crio-5ae433187aac21543251483c620755ca2c0b586fb2165594a7ad0b16dd73dacd WatchSource:0}: Error finding container 5ae433187aac21543251483c620755ca2c0b586fb2165594a7ad0b16dd73dacd: Status 404 returned error can't find the container with id 5ae433187aac21543251483c620755ca2c0b586fb2165594a7ad0b16dd73dacd
Jul 28 20:09:27.671141 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.671105    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:09:27.671666 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.671178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:09:27.675180 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.675128    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") pod \"image-registry-66788df4d7-prqht\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") " pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:09:27.677118 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.677027    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48abf100-5828-4ae4-bb41-d90680574b26-metrics-tls\") pod \"dns-default-ntjln\" (UID: \"48abf100-5828-4ae4-bb41-d90680574b26\") " pod="openshift-dns/dns-default-ntjln"
Jul 28 20:09:27.772191 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.771715    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") pod \"fd821602-a744-4e33-8e4b-5f39739c11e0\" (UID: \"fd821602-a744-4e33-8e4b-5f39739c11e0\") "
Jul 28 20:09:27.781709 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.781664    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fd821602-a744-4e33-8e4b-5f39739c11e0" (UID: "fd821602-a744-4e33-8e4b-5f39739c11e0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:09:27.872316 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.872198    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd821602-a744-4e33-8e4b-5f39739c11e0-registry-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:09:27.874153 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.874053    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-488rx\""
Jul 28 20:09:27.880990 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.880965    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ntjln"
Jul 28 20:09:27.998608 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:27.998576    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:28.075969 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.075927    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ntjln"]
Jul 28 20:09:28.208622 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.207750    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66674cb6c9-wngfz" event={"ID":"0864b1c1-412c-4667-9cbd-58ae7d7f0de3","Type":"ContainerStarted","Data":"343f3dd3bdd93d2dc30da3b764c86e893fccb7e921217a3251be7205e30e9529"}
Jul 28 20:09:28.208622 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.207817    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66674cb6c9-wngfz" event={"ID":"0864b1c1-412c-4667-9cbd-58ae7d7f0de3","Type":"ContainerStarted","Data":"5ae433187aac21543251483c620755ca2c0b586fb2165594a7ad0b16dd73dacd"}
Jul 28 20:09:28.208622 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.208563    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:28.219487 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.217515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv" event={"ID":"b4de61c9-a562-41e0-9b23-84439ed65a99","Type":"ContainerStarted","Data":"99fa8054839e916e02e8e9323bea321bd47114889a242a7c85e3fe88c77215bb"}
Jul 28 20:09:28.220491 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.220107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls" event={"ID":"d7bd0f15-554c-43eb-8052-4592d55159be","Type":"ContainerStarted","Data":"14ae6c812a0493001a921bcc109e4fcef62396421491e779b40c53fedeef3a15"}
Jul 28 20:09:28.222890 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.222484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66788df4d7-prqht"
Jul 28 20:09:28.259440 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.259352    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66674cb6c9-wngfz" podStartSLOduration=1.259330417 podStartE2EDuration="1.259330417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:28.238731199 +0000 UTC m=+67.852968332" watchObservedRunningTime="2026-07-28 20:09:28.259330417 +0000 UTC m=+67.873567550"
Jul 28 20:09:28.267536 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.267491    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66788df4d7-prqht"]
Jul 28 20:09:28.271652 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.271628    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66788df4d7-prqht"]
Jul 28 20:09:28.369417 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.369346    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:28.414195 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:09:28.414138    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48abf100_5828_4ae4_bb41_d90680574b26.slice/crio-cb23f3a872adf26e383084a610eb3effd37351361e785f6728990e976cd8d041 WatchSource:0}: Error finding container cb23f3a872adf26e383084a610eb3effd37351361e785f6728990e976cd8d041: Status 404 returned error can't find the container with id cb23f3a872adf26e383084a610eb3effd37351361e785f6728990e976cd8d041
Jul 28 20:09:28.437850 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.437800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:28.626350 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.625509    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:28.726535 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.726502    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:28.840694 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:28.840452    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd821602-a744-4e33-8e4b-5f39739c11e0" path="/var/lib/kubelet/pods/fd821602-a744-4e33-8e4b-5f39739c11e0/volumes"
Jul 28 20:09:29.234724 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:29.234593    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kmfb" event={"ID":"3ed50e77-948a-4cbd-ae6e-920b8c190c29","Type":"ContainerStarted","Data":"bb56a9f0a2ea80ed3789af5a1886a328f0f84f15e31384ea899701bb427a63dd"}
Jul 28 20:09:29.234724 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:29.234655    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kmfb" event={"ID":"3ed50e77-948a-4cbd-ae6e-920b8c190c29","Type":"ContainerStarted","Data":"ad1a404f9c1d096b6e37334eff108930b3a4918035804294b4bc92ed4fbff4ae"}
Jul 28 20:09:29.236819 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:29.236695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ntjln" event={"ID":"48abf100-5828-4ae4-bb41-d90680574b26","Type":"ContainerStarted","Data":"cb23f3a872adf26e383084a610eb3effd37351361e785f6728990e976cd8d041"}
Jul 28 20:09:29.251465 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:29.250477    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6kmfb" podStartSLOduration=66.797877466 podStartE2EDuration="1m8.250454472s" totalImagesPullingTime="1.452577006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:29.249240952 +0000 UTC m=+68.863478080" watchObservedRunningTime="2026-07-28 20:09:29.250454472 +0000 UTC m=+68.864691605"
Jul 28 20:09:29.376670 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:29.376495    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:29.558068 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:29.557949    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:29.660057 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:29.659385    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:32.807699 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:32.807663    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:33.334102 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:33.334061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:33.408973 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:33.408924    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:37.024185 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.024152    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:37.125347 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.125316    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:37.270431 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.270093    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wqnw5" event={"ID":"503db72d-49cf-4f24-9b5a-3b06f54fbbfc","Type":"ContainerStarted","Data":"a1f9097d93d1b2bbbfd9eec2c04fa4e86bb3157a884945c322915016ac39496b"}
Jul 28 20:09:37.270431 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.270384    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:09:37.272360 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.272337    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ntjln" event={"ID":"48abf100-5828-4ae4-bb41-d90680574b26","Type":"ContainerStarted","Data":"e0903affb59c3183e01a81ef01322c65b337c7ea4f6d0f27242b009c7ce60a1e"}
Jul 28 20:09:37.274897 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.274872    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv" event={"ID":"b4de61c9-a562-41e0-9b23-84439ed65a99","Type":"ContainerStarted","Data":"ae340fee54643a817807ef7960cf462c673a28c219e131bd97f1319c27435069"}
Jul 28 20:09:37.279007 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.277849    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j2kp2" event={"ID":"d953ada0-b545-41e5-901e-4b69b4d33bae","Type":"ContainerStarted","Data":"6216aa010f02fcdb15bad36e77967c44cbf8bad97360f7650ce5e80b3f2a530f"}
Jul 28 20:09:37.283598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.283550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls" event={"ID":"d7bd0f15-554c-43eb-8052-4592d55159be","Type":"ContainerStarted","Data":"cba8345a4c84f63fb566617010a6cda9459268f91372fcacb5c8666b9bc033f9"}
Jul 28 20:09:37.287555 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.287197    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wqnw5" podStartSLOduration=66.32249027 podStartE2EDuration="1m16.287179198s" totalImagesPullingTime="9.964688928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:37.285817597 +0000 UTC m=+76.900054722" watchObservedRunningTime="2026-07-28 20:09:37.287179198 +0000 UTC m=+76.901416331"
Jul 28 20:09:37.289037 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.288482    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgnhx" event={"ID":"898ae9a8-22e1-41de-90c4-ad47d2ac4e47","Type":"ContainerStarted","Data":"a66b21d40b68b30083b7bf5451731f2eab80bb1dd51daa471b8db0128de6a659"}
Jul 28 20:09:37.300564 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.300337    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-985b44966-qxtzv" podStartSLOduration=1.730090769 podStartE2EDuration="11.300319197s" totalImagesPullingTime="9.570228428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:37.299865134 +0000 UTC m=+76.914102268" watchObservedRunningTime="2026-07-28 20:09:37.300319197 +0000 UTC m=+76.914556330"
Jul 28 20:09:37.318669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.318628    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j2kp2" podStartSLOduration=66.168979613 podStartE2EDuration="1m16.318613139s" totalImagesPullingTime="10.149633526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:37.31732771 +0000 UTC m=+76.931564847" watchObservedRunningTime="2026-07-28 20:09:37.318613139 +0000 UTC m=+76.932850271"
Jul 28 20:09:37.333692 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:37.333644    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rgnhx" podStartSLOduration=17.28606844 podStartE2EDuration="36.333625582s" totalImagesPullingTime="19.047557142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:09:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:37.332758054 +0000 UTC m=+76.946995204" watchObservedRunningTime="2026-07-28 20:09:37.333625582 +0000 UTC m=+76.947862717"
Jul 28 20:09:38.294598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:38.294568    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ntjln" event={"ID":"48abf100-5828-4ae4-bb41-d90680574b26","Type":"ContainerStarted","Data":"faa06d3a47ef15f931cc98d3b93f512d0d91911326e29e5efe609ef6f19ebc2a"}
Jul 28 20:09:38.295033 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:38.294643    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ntjln"
Jul 28 20:09:38.296632 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:38.296608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls" event={"ID":"d7bd0f15-554c-43eb-8052-4592d55159be","Type":"ContainerStarted","Data":"41bd9e19299c72ff13f5cf073c9291cb7ecf67b2e76b890f04f7e374e45127a0"}
Jul 28 20:09:38.296760 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:38.296639    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls" event={"ID":"d7bd0f15-554c-43eb-8052-4592d55159be","Type":"ContainerStarted","Data":"835c4219f0841c3db3b3c942397ade47b4e08339a809117138d5eff8f7cffb04"}
Jul 28 20:09:38.311352 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:38.311311    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ntjln" podStartSLOduration=34.701939673 podStartE2EDuration="43.311295192s" totalImagesPullingTime="8.609355519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:38.310526671 +0000 UTC m=+77.924763803" watchObservedRunningTime="2026-07-28 20:09:38.311295192 +0000 UTC m=+77.925532382"
Jul 28 20:09:38.327574 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:38.327521    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c6fc98875-rgdls" podStartSLOduration=2.733398543 podStartE2EDuration="12.327504169s" totalImagesPullingTime="9.594105626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:38.326736188 +0000 UTC m=+77.940973337" watchObservedRunningTime="2026-07-28 20:09:38.327504169 +0000 UTC m=+77.941741301"
Jul 28 20:09:38.601583 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:38.601485    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:39.053510 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:39.053472    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:39.154461 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:39.154424    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:39.254462 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:39.254427    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:39.375912 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:39.375831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:39.621299 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:39.621269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:39.677871 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:39.677831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:40.810439 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:40.810387    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:40.912216 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:40.912181    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:43.757043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:43.756922    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:43.857543 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:43.857504    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:43.913615 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:43.913577    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:44.898908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:44.898875    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:44.962916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:44.962877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:45.286196 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:45.286115    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:45.357932 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:45.357900    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:45.457183 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:45.457153    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:46.006701 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:46.006672    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:46.108180 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:46.108134    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:47.373662 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:47.373628    2575 patch_prober.go:28] interesting pod/image-registry-66674cb6c9-wngfz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 20:09:47.374063 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:47.373701    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-66674cb6c9-wngfz" podUID="0864b1c1-412c-4667-9cbd-58ae7d7f0de3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:09:48.302371 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:48.302338    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ntjln"
Jul 28 20:09:49.366924 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:49.366888    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:50.246076 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:50.246043    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66674cb6c9-wngfz"
Jul 28 20:09:50.383969 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:50.383938    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:50.485178 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:50.485150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:50.571139 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:50.571064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:50.903275 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:50.903220    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:50.969515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:50.969487    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:51.521518 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.521482    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:51.633521 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.633469    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:51.820320 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.820207    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:51.895818 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.895788    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bq8qv"]
Jul 28 20:09:51.903710 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.903680    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:51.906653 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.906631    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 20:09:51.906784 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.906674    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 20:09:51.906855 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.906787    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 20:09:51.906917 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.906878    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 20:09:51.907370 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.907355    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 20:09:51.907553 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.907538    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w8qwr\""
Jul 28 20:09:51.907965 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.907949    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 20:09:51.976456 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:51.976421    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:52.087757 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.087658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-tls\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.087757 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.087732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92d3c5f9-8862-413f-8924-0ddd15bb4558-root\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.087961 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.087833    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d3c5f9-8862-413f-8924-0ddd15bb4558-sys\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.087961 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.087865    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-textfile\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.087961 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.087885    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l729m\" (UniqueName: \"kubernetes.io/projected/92d3c5f9-8862-413f-8924-0ddd15bb4558-kube-api-access-l729m\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.087961 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.087903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-wtmp\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.088124 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.087998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-accelerators-collector-config\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.088124 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.088049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.088124 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.088101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d3c5f9-8862-413f-8924-0ddd15bb4558-metrics-client-ca\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.088399 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.088375    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:52.188743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188710    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d3c5f9-8862-413f-8924-0ddd15bb4558-sys\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.188743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-textfile\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.188908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l729m\" (UniqueName: \"kubernetes.io/projected/92d3c5f9-8862-413f-8924-0ddd15bb4558-kube-api-access-l729m\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.188908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-wtmp\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.188908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188813    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d3c5f9-8862-413f-8924-0ddd15bb4558-sys\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.188908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188836    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-accelerators-collector-config\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.188908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.189175 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d3c5f9-8862-413f-8924-0ddd15bb4558-metrics-client-ca\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.189175 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-tls\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.189175 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188980    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92d3c5f9-8862-413f-8924-0ddd15bb4558-root\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.189175 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.188997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-wtmp\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.189175 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.189067    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92d3c5f9-8862-413f-8924-0ddd15bb4558-root\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.189175 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.189144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-textfile\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.189506 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.189492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-accelerators-collector-config\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.189630 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.189608    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d3c5f9-8862-413f-8924-0ddd15bb4558-metrics-client-ca\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.191306 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.191278    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.191422 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.191400    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92d3c5f9-8862-413f-8924-0ddd15bb4558-node-exporter-tls\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.198362 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.198343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l729m\" (UniqueName: \"kubernetes.io/projected/92d3c5f9-8862-413f-8924-0ddd15bb4558-kube-api-access-l729m\") pod \"node-exporter-bq8qv\" (UID: \"92d3c5f9-8862-413f-8924-0ddd15bb4558\") " pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.214575 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.214535    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bq8qv"
Jul 28 20:09:52.225561 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:09:52.225534    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d3c5f9_8862_413f_8924_0ddd15bb4558.slice/crio-2cbb3aca0edce7643db1af4f3596466d3cbf33f0067609be0e977906c9eb067c WatchSource:0}: Error finding container 2cbb3aca0edce7643db1af4f3596466d3cbf33f0067609be0e977906c9eb067c: Status 404 returned error can't find the container with id 2cbb3aca0edce7643db1af4f3596466d3cbf33f0067609be0e977906c9eb067c
Jul 28 20:09:52.346629 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.346530    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq8qv" event={"ID":"92d3c5f9-8862-413f-8924-0ddd15bb4558","Type":"ContainerStarted","Data":"2cbb3aca0edce7643db1af4f3596466d3cbf33f0067609be0e977906c9eb067c"}
Jul 28 20:09:52.347674 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.347647    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:52.520148 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.520116    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:52.667109 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.667068    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:52.762112 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.762076    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:52.859449 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:52.859415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:53.014243 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:53.014155    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:53.187653 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:53.187628    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:53.351547 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:53.351470    2575 generic.go:356] "Generic (PLEG): container finished" podID="92d3c5f9-8862-413f-8924-0ddd15bb4558" containerID="915ec83fe2bd09a61c12d32e10c44aca73bd66a65c874e17ff5bc9a12dbb4216" exitCode=0
Jul 28 20:09:53.351547 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:53.351514    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq8qv" event={"ID":"92d3c5f9-8862-413f-8924-0ddd15bb4558","Type":"ContainerDied","Data":"915ec83fe2bd09a61c12d32e10c44aca73bd66a65c874e17ff5bc9a12dbb4216"}
Jul 28 20:09:54.357479 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:54.357444    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq8qv" event={"ID":"92d3c5f9-8862-413f-8924-0ddd15bb4558","Type":"ContainerStarted","Data":"d9007a649766a4a7e7449e131f3412f0e386e6f80d8be06692bb563605fad483"}
Jul 28 20:09:54.357939 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:54.357503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq8qv" event={"ID":"92d3c5f9-8862-413f-8924-0ddd15bb4558","Type":"ContainerStarted","Data":"303f206175205908595f7d94501229d994458afb85cbd94f0d6ed6a3d85f053a"}
Jul 28 20:09:54.378502 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:54.378451    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bq8qv" podStartSLOduration=2.421256479 podStartE2EDuration="3.378433245s" totalImagesPullingTime="890.431355ms" totalInitContainerRuntime="66.745411ms" isStatefulPod=true podCreationTimestamp="2026-07-28 20:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:54.377682201 +0000 UTC m=+93.991919335" watchObservedRunningTime="2026-07-28 20:09:54.378433245 +0000 UTC m=+93.992670377"
Jul 28 20:09:54.744015 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:54.743979    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:54.828660 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:54.828626    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:55.023246 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:55.023156    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:55.257722 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:55.257688    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:55.617730 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:55.617697    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:55.719928 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:55.719897    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:56.530158 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:56.530117    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:56.607370 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:56.607339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:57.629060 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:57.629024    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:57.692874 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:57.692841    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:58.099402 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:58.099373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:58.255463 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:58.255428    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:58.356551 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:09:58.356472    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:02.151593 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:02.151554    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:02.250757 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:02.250726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:03.122339 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:03.122300    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:03.221243 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:03.221193    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:03.657940 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:03.657904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:06.480772 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:06.480732    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:06.588329 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:06.588295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:07.039578 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:07.039539    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:07.193149 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:07.193114    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:08.299187 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:08.299153    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wqnw5"
Jul 28 20:10:11.262702 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:11.262673    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:11.361322 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:11.361289    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:13.850061 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:13.850022    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:13.950010 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:13.949975    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:14.050151 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:14.050118    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:22.042956 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:22.042919    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:22.152289 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:22.152256    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:29.653810 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:29.653776    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:29.758721 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:29.758687    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:31.233375 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:31.233347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:32.039662 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:32.039629    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:32.100537 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:32.100512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:37.029134 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:37.029099    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:37.092754 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:37.092727    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:37.380792 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:37.380717    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:37.473932 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:37.473896    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:44.101469 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:44.101438    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:44.194139 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:10:44.194108    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:07.915904 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:11:07.915865    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:07.981736 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:11:07.981701    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:15.916747 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:11:15.916667    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:17.733706 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:11:17.733671    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:19.911045 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:11:19.911014    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:19.975335 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:11:19.975298    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:27.116455 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:11:27.116420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:37.046797 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:11:37.046758    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:37.148386 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:11:37.148352    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:02.141594 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:02.141561    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:07.453005 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:07.452970    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:07.552355 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:07.552323    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:22.299393 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:22.299355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:22.394821 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:22.394780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:49.444249 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:49.444159    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:50.815802 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:50.815769    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:53.047814 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:53.047770    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:53.158879 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:53.158840    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:55.152156 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:55.152124    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:55.249542 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:55.249505    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:55.760469 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:55.760440    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:55.861947 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:55.861917    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:57.150076 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:57.150041    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:57.537455 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:57.537383    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:59.835515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:12:59.835485    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:00.505141 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:00.505110    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:02.702213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:02.702174    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:02.771211 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:02.771177    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:05.748244 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:05.748182    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:07.255276 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:07.255243    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:07.354074 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:07.354042    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:14.526446 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:14.526409    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:14.629326 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:14.629291    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:15.848334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:15.848295    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ckm6x"]
Jul 28 20:13:15.851485 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:15.851462    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ckm6x"
Jul 28 20:13:15.853730 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:15.853707    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 20:13:15.854080 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:15.854064    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 20:13:15.854555 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:15.854540    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-95ppb\""
Jul 28 20:13:15.862494 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:15.862473    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ckm6x"]
Jul 28 20:13:15.934148 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:15.934112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wrrs\" (UniqueName: \"kubernetes.io/projected/62836777-4dcd-49f3-a12b-b4881e7f8ffa-kube-api-access-8wrrs\") pod \"cert-manager-55b4d954d5-ckm6x\" (UID: \"62836777-4dcd-49f3-a12b-b4881e7f8ffa\") " pod="cert-manager/cert-manager-55b4d954d5-ckm6x"
Jul 28 20:13:15.934356 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:15.934166    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62836777-4dcd-49f3-a12b-b4881e7f8ffa-bound-sa-token\") pod \"cert-manager-55b4d954d5-ckm6x\" (UID: \"62836777-4dcd-49f3-a12b-b4881e7f8ffa\") " pod="cert-manager/cert-manager-55b4d954d5-ckm6x"
Jul 28 20:13:16.034935 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:16.034902    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrrs\" (UniqueName: \"kubernetes.io/projected/62836777-4dcd-49f3-a12b-b4881e7f8ffa-kube-api-access-8wrrs\") pod \"cert-manager-55b4d954d5-ckm6x\" (UID: \"62836777-4dcd-49f3-a12b-b4881e7f8ffa\") " pod="cert-manager/cert-manager-55b4d954d5-ckm6x"
Jul 28 20:13:16.035114 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:16.034952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62836777-4dcd-49f3-a12b-b4881e7f8ffa-bound-sa-token\") pod \"cert-manager-55b4d954d5-ckm6x\" (UID: \"62836777-4dcd-49f3-a12b-b4881e7f8ffa\") " pod="cert-manager/cert-manager-55b4d954d5-ckm6x"
Jul 28 20:13:16.042993 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:16.042961    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrrs\" (UniqueName: \"kubernetes.io/projected/62836777-4dcd-49f3-a12b-b4881e7f8ffa-kube-api-access-8wrrs\") pod \"cert-manager-55b4d954d5-ckm6x\" (UID: \"62836777-4dcd-49f3-a12b-b4881e7f8ffa\") " pod="cert-manager/cert-manager-55b4d954d5-ckm6x"
Jul 28 20:13:16.043452 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:16.043435    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62836777-4dcd-49f3-a12b-b4881e7f8ffa-bound-sa-token\") pod \"cert-manager-55b4d954d5-ckm6x\" (UID: \"62836777-4dcd-49f3-a12b-b4881e7f8ffa\") " pod="cert-manager/cert-manager-55b4d954d5-ckm6x"
Jul 28 20:13:16.159959 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:16.159926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ckm6x"
Jul 28 20:13:16.265895 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:16.265864    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:16.286851 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:16.286822    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ckm6x"]
Jul 28 20:13:16.289746 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:13:16.289713    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62836777_4dcd_49f3_a12b_b4881e7f8ffa.slice/crio-573a02b065853deb8ecfd57e0f0d417d2fba9d351fe9053da5358f8827fae577 WatchSource:0}: Error finding container 573a02b065853deb8ecfd57e0f0d417d2fba9d351fe9053da5358f8827fae577: Status 404 returned error can't find the container with id 573a02b065853deb8ecfd57e0f0d417d2fba9d351fe9053da5358f8827fae577
Jul 28 20:13:17.017310 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:17.017255    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ckm6x" event={"ID":"62836777-4dcd-49f3-a12b-b4881e7f8ffa","Type":"ContainerStarted","Data":"573a02b065853deb8ecfd57e0f0d417d2fba9d351fe9053da5358f8827fae577"}
Jul 28 20:13:20.029432 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:20.029390    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ckm6x" event={"ID":"62836777-4dcd-49f3-a12b-b4881e7f8ffa","Type":"ContainerStarted","Data":"a46a4a7919d7ac45797d4d76f5c7d0cfe54f28b1c8358bf15ff9f32c06bbf703"}
Jul 28 20:13:20.045676 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:20.045624    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ckm6x" podStartSLOduration=2.106267016 podStartE2EDuration="5.045608777s" totalImagesPullingTime="2.939341761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:13:20.045006718 +0000 UTC m=+299.659243850" watchObservedRunningTime="2026-07-28 20:13:20.045608777 +0000 UTC m=+299.659845910"
Jul 28 20:13:20.745956 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:20.745928    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 20:13:22.557381 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:22.557342    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:24.679204 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:24.679162    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:28.280870 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:28.280837    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:28.376959 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:28.376922    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:28.862801 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:28.862769    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:29.123222 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.123145    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:29.238582 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.238548    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:29.281700 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.281664    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"]
Jul 28 20:13:29.284933 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.284914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.288653 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.288629    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 20:13:29.288780 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.288653    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h57cb\""
Jul 28 20:13:29.288780 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.288682    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 20:13:29.288780 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.288732    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 20:13:29.288780 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.288743    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 20:13:29.314842 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.314803    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"]
Jul 28 20:13:29.438952 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.438908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7a4a1da-4f55-4800-b382-2291328d3740-tmp\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.438952 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.438956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdltc\" (UniqueName: \"kubernetes.io/projected/c7a4a1da-4f55-4800-b382-2291328d3740-kube-api-access-hdltc\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.439189 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.438981    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c7a4a1da-4f55-4800-b382-2291328d3740-manifests\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.439189 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.439036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c7a4a1da-4f55-4800-b382-2291328d3740-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.439189 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.439053    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7a4a1da-4f55-4800-b382-2291328d3740-webhook-cert\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.447894 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.447866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:29.540119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.540080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7a4a1da-4f55-4800-b382-2291328d3740-tmp\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.540309 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.540129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdltc\" (UniqueName: \"kubernetes.io/projected/c7a4a1da-4f55-4800-b382-2291328d3740-kube-api-access-hdltc\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.540309 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.540263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c7a4a1da-4f55-4800-b382-2291328d3740-manifests\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.540410 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.540316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c7a4a1da-4f55-4800-b382-2291328d3740-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.540410 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.540343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7a4a1da-4f55-4800-b382-2291328d3740-webhook-cert\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.540635 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.540609    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c7a4a1da-4f55-4800-b382-2291328d3740-manifests\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.542416 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.542387    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7a4a1da-4f55-4800-b382-2291328d3740-tmp\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.542685 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.542664    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c7a4a1da-4f55-4800-b382-2291328d3740-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.542813 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.542792    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7a4a1da-4f55-4800-b382-2291328d3740-webhook-cert\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.552855 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.552832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdltc\" (UniqueName: \"kubernetes.io/projected/c7a4a1da-4f55-4800-b382-2291328d3740-kube-api-access-hdltc\") pod \"opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv\" (UID: \"c7a4a1da-4f55-4800-b382-2291328d3740\") " pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.574941 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.574900    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:29.594892 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.594862    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:29.678753 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.678724    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:29.728161 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.728085    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"]
Jul 28 20:13:29.731023 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:13:29.730997    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a4a1da_4f55_4800_b382_2291328d3740.slice/crio-14456edf592f4ab469bddb126000a9846e6c7acf98e6b261ac104f8ddfd7ac4c WatchSource:0}: Error finding container 14456edf592f4ab469bddb126000a9846e6c7acf98e6b261ac104f8ddfd7ac4c: Status 404 returned error can't find the container with id 14456edf592f4ab469bddb126000a9846e6c7acf98e6b261ac104f8ddfd7ac4c
Jul 28 20:13:29.732845 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.732825    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:13:29.755784 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.755752    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:29.824749 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.824715    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:29.930036 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:29.929999    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:30.065727 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:30.065638    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv" event={"ID":"c7a4a1da-4f55-4800-b382-2291328d3740","Type":"ContainerStarted","Data":"14456edf592f4ab469bddb126000a9846e6c7acf98e6b261ac104f8ddfd7ac4c"}
Jul 28 20:13:33.078294 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:33.078261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv" event={"ID":"c7a4a1da-4f55-4800-b382-2291328d3740","Type":"ContainerStarted","Data":"1f6fb47081ae1fb3a32a35ca71076ae16818caf235bad74a8ddc02d98259292e"}
Jul 28 20:13:34.083780 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:34.083737    2575 generic.go:356] "Generic (PLEG): container finished" podID="c7a4a1da-4f55-4800-b382-2291328d3740" containerID="1f6fb47081ae1fb3a32a35ca71076ae16818caf235bad74a8ddc02d98259292e" exitCode=0
Jul 28 20:13:34.084161 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:34.083844    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv" event={"ID":"c7a4a1da-4f55-4800-b382-2291328d3740","Type":"ContainerDied","Data":"1f6fb47081ae1fb3a32a35ca71076ae16818caf235bad74a8ddc02d98259292e"}
Jul 28 20:13:35.088596 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:35.088556    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv" event={"ID":"c7a4a1da-4f55-4800-b382-2291328d3740","Type":"ContainerStarted","Data":"92d9ffa4073d8aefb57e4101c376b316c5035cfdc68327be88e660c2a91acd75"}
Jul 28 20:13:35.088963 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:35.088663    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:35.114899 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:35.114844    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv" podStartSLOduration=1.4204081849999999 podStartE2EDuration="6.114828078s" totalImagesPullingTime="3.604570036s" totalInitContainerRuntime="1.089849857s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:13:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:13:35.114126154 +0000 UTC m=+314.728363281" watchObservedRunningTime="2026-07-28 20:13:35.114828078 +0000 UTC m=+314.729065210"
Jul 28 20:13:46.093841 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:46.093810    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv"
Jul 28 20:13:48.509515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:48.509471    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:48.667527 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:48.667476    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:49.030008 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.029970    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:49.080127 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.080091    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"]
Jul 28 20:13:49.105280 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.105246    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"]
Jul 28 20:13:49.105455 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.105371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.109112 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.109088    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 20:13:49.109317 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.109216    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 20:13:49.109419 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.109287    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 20:13:49.109419 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.109392    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 20:13:49.110122 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.110102    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-v72ch\""
Jul 28 20:13:49.189867 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.189834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a11572-84d3-4acd-9824-243c2757d16f-tls-certs\") pod \"kube-auth-proxy-5bb564c89-gwmd4\" (UID: \"59a11572-84d3-4acd-9824-243c2757d16f\") " pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.189867 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.189872    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkgf8\" (UniqueName: \"kubernetes.io/projected/59a11572-84d3-4acd-9824-243c2757d16f-kube-api-access-vkgf8\") pod \"kube-auth-proxy-5bb564c89-gwmd4\" (UID: \"59a11572-84d3-4acd-9824-243c2757d16f\") " pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.190097 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.189902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59a11572-84d3-4acd-9824-243c2757d16f-tmp\") pod \"kube-auth-proxy-5bb564c89-gwmd4\" (UID: \"59a11572-84d3-4acd-9824-243c2757d16f\") " pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.291151 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.291067    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a11572-84d3-4acd-9824-243c2757d16f-tls-certs\") pod \"kube-auth-proxy-5bb564c89-gwmd4\" (UID: \"59a11572-84d3-4acd-9824-243c2757d16f\") " pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.291151 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.291107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkgf8\" (UniqueName: \"kubernetes.io/projected/59a11572-84d3-4acd-9824-243c2757d16f-kube-api-access-vkgf8\") pod \"kube-auth-proxy-5bb564c89-gwmd4\" (UID: \"59a11572-84d3-4acd-9824-243c2757d16f\") " pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.291151 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.291132    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59a11572-84d3-4acd-9824-243c2757d16f-tmp\") pod \"kube-auth-proxy-5bb564c89-gwmd4\" (UID: \"59a11572-84d3-4acd-9824-243c2757d16f\") " pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.291458 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:13:49.291240    2575 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 28 20:13:49.291458 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:13:49.291312    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a11572-84d3-4acd-9824-243c2757d16f-tls-certs podName:59a11572-84d3-4acd-9824-243c2757d16f nodeName:}" failed. No retries permitted until 2026-07-28 20:13:49.791292184 +0000 UTC m=+329.405529302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/59a11572-84d3-4acd-9824-243c2757d16f-tls-certs") pod "kube-auth-proxy-5bb564c89-gwmd4" (UID: "59a11572-84d3-4acd-9824-243c2757d16f") : secret "kube-auth-proxy-tls" not found
Jul 28 20:13:49.293395 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.293365    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59a11572-84d3-4acd-9824-243c2757d16f-tmp\") pod \"kube-auth-proxy-5bb564c89-gwmd4\" (UID: \"59a11572-84d3-4acd-9824-243c2757d16f\") " pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.307922 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.307897    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g7mnm"]
Jul 28 20:13:49.310785 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.310759    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkgf8\" (UniqueName: \"kubernetes.io/projected/59a11572-84d3-4acd-9824-243c2757d16f-kube-api-access-vkgf8\") pod \"kube-auth-proxy-5bb564c89-gwmd4\" (UID: \"59a11572-84d3-4acd-9824-243c2757d16f\") " pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.336469 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.336408    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g7mnm"]
Jul 28 20:13:49.336639 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.336529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:13:49.339035 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.339014    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 28 20:13:49.339369 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.339353    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6zqbp\""
Jul 28 20:13:49.363996 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.363956    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:49.392184 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.392147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3f12b87-4fc9-4e03-b35f-cc1daac7e73e-cert\") pod \"odh-model-controller-7998c67dc4-g7mnm\" (UID: \"b3f12b87-4fc9-4e03-b35f-cc1daac7e73e\") " pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:13:49.392398 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.392304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgcg\" (UniqueName: \"kubernetes.io/projected/b3f12b87-4fc9-4e03-b35f-cc1daac7e73e-kube-api-access-ldgcg\") pod \"odh-model-controller-7998c67dc4-g7mnm\" (UID: \"b3f12b87-4fc9-4e03-b35f-cc1daac7e73e\") " pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:13:49.465161 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.465126    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:49.493453 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.493415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgcg\" (UniqueName: \"kubernetes.io/projected/b3f12b87-4fc9-4e03-b35f-cc1daac7e73e-kube-api-access-ldgcg\") pod \"odh-model-controller-7998c67dc4-g7mnm\" (UID: \"b3f12b87-4fc9-4e03-b35f-cc1daac7e73e\") " pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:13:49.493453 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.493465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3f12b87-4fc9-4e03-b35f-cc1daac7e73e-cert\") pod \"odh-model-controller-7998c67dc4-g7mnm\" (UID: \"b3f12b87-4fc9-4e03-b35f-cc1daac7e73e\") " pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:13:49.493731 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:13:49.493591    2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 20:13:49.493731 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:13:49.493654    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f12b87-4fc9-4e03-b35f-cc1daac7e73e-cert podName:b3f12b87-4fc9-4e03-b35f-cc1daac7e73e nodeName:}" failed. No retries permitted until 2026-07-28 20:13:49.993636991 +0000 UTC m=+329.607874105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b3f12b87-4fc9-4e03-b35f-cc1daac7e73e-cert") pod "odh-model-controller-7998c67dc4-g7mnm" (UID: "b3f12b87-4fc9-4e03-b35f-cc1daac7e73e") : secret "odh-model-controller-webhook-cert" not found
Jul 28 20:13:49.503024 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.502992    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgcg\" (UniqueName: \"kubernetes.io/projected/b3f12b87-4fc9-4e03-b35f-cc1daac7e73e-kube-api-access-ldgcg\") pod \"odh-model-controller-7998c67dc4-g7mnm\" (UID: \"b3f12b87-4fc9-4e03-b35f-cc1daac7e73e\") " pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:13:49.565968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.565871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:49.666589 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.666529    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:49.769767 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.769731    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:49.796079 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.796043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a11572-84d3-4acd-9824-243c2757d16f-tls-certs\") pod \"kube-auth-proxy-5bb564c89-gwmd4\" (UID: \"59a11572-84d3-4acd-9824-243c2757d16f\") " pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.798390 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.798366    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a11572-84d3-4acd-9824-243c2757d16f-tls-certs\") pod \"kube-auth-proxy-5bb564c89-gwmd4\" (UID: \"59a11572-84d3-4acd-9824-243c2757d16f\") " pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:49.997782 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:49.997741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3f12b87-4fc9-4e03-b35f-cc1daac7e73e-cert\") pod \"odh-model-controller-7998c67dc4-g7mnm\" (UID: \"b3f12b87-4fc9-4e03-b35f-cc1daac7e73e\") " pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:13:50.000215 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.000188    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3f12b87-4fc9-4e03-b35f-cc1daac7e73e-cert\") pod \"odh-model-controller-7998c67dc4-g7mnm\" (UID: \"b3f12b87-4fc9-4e03-b35f-cc1daac7e73e\") " pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:13:50.019092 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.019066    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"
Jul 28 20:13:50.071508 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.071476    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:50.104588 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.104557    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lb5h7"]
Jul 28 20:13:50.120379 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.120354    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lb5h7"]
Jul 28 20:13:50.120468 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.120412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:13:50.124739 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.124680    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 28 20:13:50.124893 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.124871    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-jswrz\""
Jul 28 20:13:50.185683 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.185654    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4"]
Jul 28 20:13:50.188039 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:13:50.188012    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a11572_84d3_4acd_9824_243c2757d16f.slice/crio-4cfdbb4d438ac41aa7d1ef417a7f671f6ae023c153117a3f26f192e8ca20b03b WatchSource:0}: Error finding container 4cfdbb4d438ac41aa7d1ef417a7f671f6ae023c153117a3f26f192e8ca20b03b: Status 404 returned error can't find the container with id 4cfdbb4d438ac41aa7d1ef417a7f671f6ae023c153117a3f26f192e8ca20b03b
Jul 28 20:13:50.199717 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.199689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ee841f-e97b-4f95-b29b-e6a7fc27965d-tls-certs\") pod \"model-serving-api-54d89bd5fc-lb5h7\" (UID: \"59ee841f-e97b-4f95-b29b-e6a7fc27965d\") " pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:13:50.199827 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.199807    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s54v\" (UniqueName: \"kubernetes.io/projected/59ee841f-e97b-4f95-b29b-e6a7fc27965d-kube-api-access-2s54v\") pod \"model-serving-api-54d89bd5fc-lb5h7\" (UID: \"59ee841f-e97b-4f95-b29b-e6a7fc27965d\") " pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:13:50.251661 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.251567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:13:50.300276 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.300222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s54v\" (UniqueName: \"kubernetes.io/projected/59ee841f-e97b-4f95-b29b-e6a7fc27965d-kube-api-access-2s54v\") pod \"model-serving-api-54d89bd5fc-lb5h7\" (UID: \"59ee841f-e97b-4f95-b29b-e6a7fc27965d\") " pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:13:50.300447 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.300306    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ee841f-e97b-4f95-b29b-e6a7fc27965d-tls-certs\") pod \"model-serving-api-54d89bd5fc-lb5h7\" (UID: \"59ee841f-e97b-4f95-b29b-e6a7fc27965d\") " pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:13:50.300447 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:13:50.300417    2575 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 20:13:50.300552 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:13:50.300482    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ee841f-e97b-4f95-b29b-e6a7fc27965d-tls-certs podName:59ee841f-e97b-4f95-b29b-e6a7fc27965d nodeName:}" failed. No retries permitted until 2026-07-28 20:13:50.80046282 +0000 UTC m=+330.414699932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/59ee841f-e97b-4f95-b29b-e6a7fc27965d-tls-certs") pod "model-serving-api-54d89bd5fc-lb5h7" (UID: "59ee841f-e97b-4f95-b29b-e6a7fc27965d") : secret "model-serving-api-tls" not found
Jul 28 20:13:50.322594 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.322560    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s54v\" (UniqueName: \"kubernetes.io/projected/59ee841f-e97b-4f95-b29b-e6a7fc27965d-kube-api-access-2s54v\") pod \"model-serving-api-54d89bd5fc-lb5h7\" (UID: \"59ee841f-e97b-4f95-b29b-e6a7fc27965d\") " pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:13:50.427091 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.427062    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g7mnm"]
Jul 28 20:13:50.670831 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.670792    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:50.677903 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.677865    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"]
Jul 28 20:13:50.680735 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.680712    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.684884 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.684854    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8nbz4\""
Jul 28 20:13:50.685010 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.684970    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 28 20:13:50.685187 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.685167    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 28 20:13:50.685285 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.685215    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 20:13:50.685349 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.685335    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 28 20:13:50.701553 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.701526    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"]
Jul 28 20:13:50.703802 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.703781    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d08bc1-7c6f-43d9-887c-e904c0561ce9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.703905 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.703811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.703905 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.703838    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.703905 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.703870    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.703905 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.703902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbtsb\" (UniqueName: \"kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-kube-api-access-xbtsb\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.704037 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.703919    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.704037 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.703989    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.766122 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.766084    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:50.805190 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.805157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d08bc1-7c6f-43d9-887c-e904c0561ce9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.805402 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.805199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.805402 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.805250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.805402 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.805281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.805402 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.805313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbtsb\" (UniqueName: \"kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-kube-api-access-xbtsb\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.805402 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.805344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.805402 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.805378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.805656 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.805436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ee841f-e97b-4f95-b29b-e6a7fc27965d-tls-certs\") pod \"model-serving-api-54d89bd5fc-lb5h7\" (UID: \"59ee841f-e97b-4f95-b29b-e6a7fc27965d\") " pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:13:50.806554 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.806528    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.808062 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.808025    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d08bc1-7c6f-43d9-887c-e904c0561ce9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.808341 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.808313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.808417 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.808401    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.808468 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.808439    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.808954 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.808931    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ee841f-e97b-4f95-b29b-e6a7fc27965d-tls-certs\") pod \"model-serving-api-54d89bd5fc-lb5h7\" (UID: \"59ee841f-e97b-4f95-b29b-e6a7fc27965d\") " pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:13:50.814028 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.813994    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbtsb\" (UniqueName: \"kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-kube-api-access-xbtsb\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.814130 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.814047    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7pdmk\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:50.866997 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.866958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:50.990072 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:50.989988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:51.040542 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:51.040452    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:13:51.155599 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:51.153488    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm" event={"ID":"b3f12b87-4fc9-4e03-b35f-cc1daac7e73e","Type":"ContainerStarted","Data":"70650fdb914137e427fe4996b6d03ec414f3e319fafee94c126ee7e2d746da5c"}
Jul 28 20:13:51.155599 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:51.155196    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4" event={"ID":"59a11572-84d3-4acd-9824-243c2757d16f","Type":"ContainerStarted","Data":"4cfdbb4d438ac41aa7d1ef417a7f671f6ae023c153117a3f26f192e8ca20b03b"}
Jul 28 20:13:51.168964 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:51.163419    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"]
Jul 28 20:13:51.239023 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:51.238876    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lb5h7"]
Jul 28 20:13:51.244623 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:13:51.244589    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ee841f_e97b_4f95_b29b_e6a7fc27965d.slice/crio-b2c02a6b6ddc508e7692940381bb6dd799ace830735d58beff9c7cd0d2378f84 WatchSource:0}: Error finding container b2c02a6b6ddc508e7692940381bb6dd799ace830735d58beff9c7cd0d2378f84: Status 404 returned error can't find the container with id b2c02a6b6ddc508e7692940381bb6dd799ace830735d58beff9c7cd0d2378f84
Jul 28 20:13:51.822535 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:51.822493    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:52.164503 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:52.164405    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7" event={"ID":"59ee841f-e97b-4f95-b29b-e6a7fc27965d","Type":"ContainerStarted","Data":"b2c02a6b6ddc508e7692940381bb6dd799ace830735d58beff9c7cd0d2378f84"}
Jul 28 20:13:52.166908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:52.166823    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk" event={"ID":"17d08bc1-7c6f-43d9-887c-e904c0561ce9","Type":"ContainerStarted","Data":"d7b718843fb84cbad00814113227cc25769c9418029f1c5578bc1d2b8022f849"}
Jul 28 20:13:55.794551 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:55.794500    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 20:13:55.794931 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:55.794590    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 20:13:56.191090 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.191052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm" event={"ID":"b3f12b87-4fc9-4e03-b35f-cc1daac7e73e","Type":"ContainerStarted","Data":"ee15eb1418e9647d5170c6209ba14c6dabaf6910f1ae3267284038f964d5be3b"}
Jul 28 20:13:56.191294 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.191174    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:13:56.192736 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.192706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7" event={"ID":"59ee841f-e97b-4f95-b29b-e6a7fc27965d","Type":"ContainerStarted","Data":"b2b0090aaf796fc7c4d2b0c59c13879fdfb6252051cf74fdeb784b817c68c833"}
Jul 28 20:13:56.192868 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.192757    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:13:56.194282 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.194260    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk" event={"ID":"17d08bc1-7c6f-43d9-887c-e904c0561ce9","Type":"ContainerStarted","Data":"f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e"}
Jul 28 20:13:56.194421 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.194404    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:13:56.195965 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.195939    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4" event={"ID":"59a11572-84d3-4acd-9824-243c2757d16f","Type":"ContainerStarted","Data":"dd1ad8eac4fdba865eb4543188b54baf915fafab57373e4e0298ebb82beef045"}
Jul 28 20:13:56.196391 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.196368    2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-7pdmk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:13:56.196479 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.196419    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk" podUID="17d08bc1-7c6f-43d9-887c-e904c0561ce9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:13:56.210320 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.210273    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm" podStartSLOduration=1.8417230949999999 podStartE2EDuration="7.210258014s" totalImagesPullingTime="5.368534919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:13:56.209039626 +0000 UTC m=+335.823276762" watchObservedRunningTime="2026-07-28 20:13:56.210258014 +0000 UTC m=+335.824495147"
Jul 28 20:13:56.227510 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.227463    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bb564c89-gwmd4" podStartSLOduration=1.6276214850000001 podStartE2EDuration="7.22744739s" totalImagesPullingTime="5.599825905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:13:56.22594388 +0000 UTC m=+335.840181014" watchObservedRunningTime="2026-07-28 20:13:56.22744739 +0000 UTC m=+335.841684516"
Jul 28 20:13:56.253734 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.253680    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk" podStartSLOduration=1.627000947 podStartE2EDuration="6.253657634s" totalImagesPullingTime="4.626656687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:13:56.253027034 +0000 UTC m=+335.867264160" watchObservedRunningTime="2026-07-28 20:13:56.253657634 +0000 UTC m=+335.867894768"
Jul 28 20:13:56.273642 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.273591    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7" podStartSLOduration=1.73058114 podStartE2EDuration="6.273569557s" totalImagesPullingTime="4.542988417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:13:56.272742318 +0000 UTC m=+335.886979451" watchObservedRunningTime="2026-07-28 20:13:56.273569557 +0000 UTC m=+335.887806692"
Jul 28 20:13:56.834365 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:56.834329    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:57.132011 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:57.131900    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:57.201478 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:57.201447    2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-7pdmk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:13:57.201672 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:57.201515    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk" podUID="17d08bc1-7c6f-43d9-887c-e904c0561ce9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:13:57.654351 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:57.654313    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:57.915072 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:57.914998    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:58.015520 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:13:58.015487    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:00.201752 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:00.201721    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:14:00.951620 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:00.951588    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:01.049859 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.049757    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:01.055266 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.055237    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"]
Jul 28 20:14:01.058564 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.058546    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:01.062598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.062579    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vc7ms\""
Jul 28 20:14:01.062740 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.062722    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 28 20:14:01.066897 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.066871    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"]
Jul 28 20:14:01.070474 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.070451    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"
Jul 28 20:14:01.070726 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.070690    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"]
Jul 28 20:14:01.074942 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.074925    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vw8tn\""
Jul 28 20:14:01.075386 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.075367    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 28 20:14:01.088052 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.087958    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"]
Jul 28 20:14:01.145335 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.145301    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:01.195635 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.195597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37e161ce-b65e-42a3-ab4a-a669bb4a3880-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h677p\" (UID: \"37e161ce-b65e-42a3-ab4a-a669bb4a3880\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"
Jul 28 20:14:01.195846 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.195651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg2sz\" (UniqueName: \"kubernetes.io/projected/37e161ce-b65e-42a3-ab4a-a669bb4a3880-kube-api-access-xg2sz\") pod \"llmisvc-controller-manager-777c6bd7d5-h677p\" (UID: \"37e161ce-b65e-42a3-ab4a-a669bb4a3880\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"
Jul 28 20:14:01.195846 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.195790    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c3fde1b-a971-4276-9a68-a304440d6094-cert\") pod \"kserve-controller-manager-d6f4b65b8-cqcqj\" (UID: \"3c3fde1b-a971-4276-9a68-a304440d6094\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:01.195846 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.195828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x4b8\" (UniqueName: \"kubernetes.io/projected/3c3fde1b-a971-4276-9a68-a304440d6094-kube-api-access-6x4b8\") pod \"kserve-controller-manager-d6f4b65b8-cqcqj\" (UID: \"3c3fde1b-a971-4276-9a68-a304440d6094\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:01.282183 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.282147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:01.296395 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.296314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37e161ce-b65e-42a3-ab4a-a669bb4a3880-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h677p\" (UID: \"37e161ce-b65e-42a3-ab4a-a669bb4a3880\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"
Jul 28 20:14:01.296395 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.296390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg2sz\" (UniqueName: \"kubernetes.io/projected/37e161ce-b65e-42a3-ab4a-a669bb4a3880-kube-api-access-xg2sz\") pod \"llmisvc-controller-manager-777c6bd7d5-h677p\" (UID: \"37e161ce-b65e-42a3-ab4a-a669bb4a3880\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"
Jul 28 20:14:01.296639 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.296454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c3fde1b-a971-4276-9a68-a304440d6094-cert\") pod \"kserve-controller-manager-d6f4b65b8-cqcqj\" (UID: \"3c3fde1b-a971-4276-9a68-a304440d6094\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:01.296639 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.296500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x4b8\" (UniqueName: \"kubernetes.io/projected/3c3fde1b-a971-4276-9a68-a304440d6094-kube-api-access-6x4b8\") pod \"kserve-controller-manager-d6f4b65b8-cqcqj\" (UID: \"3c3fde1b-a971-4276-9a68-a304440d6094\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:01.296890 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:14:01.296864    2575 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 28 20:14:01.297022 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:14:01.296941    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c3fde1b-a971-4276-9a68-a304440d6094-cert podName:3c3fde1b-a971-4276-9a68-a304440d6094 nodeName:}" failed. No retries permitted until 2026-07-28 20:14:01.7969199 +0000 UTC m=+341.411157017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3c3fde1b-a971-4276-9a68-a304440d6094-cert") pod "kserve-controller-manager-d6f4b65b8-cqcqj" (UID: "3c3fde1b-a971-4276-9a68-a304440d6094") : secret "kserve-webhook-server-cert" not found
Jul 28 20:14:01.298919 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.298897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37e161ce-b65e-42a3-ab4a-a669bb4a3880-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h677p\" (UID: \"37e161ce-b65e-42a3-ab4a-a669bb4a3880\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"
Jul 28 20:14:01.313120 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.313056    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg2sz\" (UniqueName: \"kubernetes.io/projected/37e161ce-b65e-42a3-ab4a-a669bb4a3880-kube-api-access-xg2sz\") pod \"llmisvc-controller-manager-777c6bd7d5-h677p\" (UID: \"37e161ce-b65e-42a3-ab4a-a669bb4a3880\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"
Jul 28 20:14:01.313120 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.313111    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x4b8\" (UniqueName: \"kubernetes.io/projected/3c3fde1b-a971-4276-9a68-a304440d6094-kube-api-access-6x4b8\") pod \"kserve-controller-manager-d6f4b65b8-cqcqj\" (UID: \"3c3fde1b-a971-4276-9a68-a304440d6094\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:01.382908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.382864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"
Jul 28 20:14:01.460928 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.460868    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:01.562057 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.562016    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"]
Jul 28 20:14:01.565620 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:14:01.565549    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod37e161ce_b65e_42a3_ab4a_a669bb4a3880.slice/crio-fbfd029e8066a819ce7f8d61ad86a437d4d26447c53312674a66ea40f05ee772 WatchSource:0}: Error finding container fbfd029e8066a819ce7f8d61ad86a437d4d26447c53312674a66ea40f05ee772: Status 404 returned error can't find the container with id fbfd029e8066a819ce7f8d61ad86a437d4d26447c53312674a66ea40f05ee772
Jul 28 20:14:01.636375 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.636340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:01.742740 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.742696    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:01.799908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.799871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c3fde1b-a971-4276-9a68-a304440d6094-cert\") pod \"kserve-controller-manager-d6f4b65b8-cqcqj\" (UID: \"3c3fde1b-a971-4276-9a68-a304440d6094\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:01.802555 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.802531    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c3fde1b-a971-4276-9a68-a304440d6094-cert\") pod \"kserve-controller-manager-d6f4b65b8-cqcqj\" (UID: \"3c3fde1b-a971-4276-9a68-a304440d6094\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:01.968760 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:01.968727    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:02.120762 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:02.120712    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"]
Jul 28 20:14:02.123588 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:14:02.123563    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3fde1b_a971_4276_9a68_a304440d6094.slice/crio-dc19ab3758007dcf04004d1e56e6846653c7fd0f9c337120901218ac42203a0d WatchSource:0}: Error finding container dc19ab3758007dcf04004d1e56e6846653c7fd0f9c337120901218ac42203a0d: Status 404 returned error can't find the container with id dc19ab3758007dcf04004d1e56e6846653c7fd0f9c337120901218ac42203a0d
Jul 28 20:14:02.222771 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:02.222687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj" event={"ID":"3c3fde1b-a971-4276-9a68-a304440d6094","Type":"ContainerStarted","Data":"dc19ab3758007dcf04004d1e56e6846653c7fd0f9c337120901218ac42203a0d"}
Jul 28 20:14:02.223765 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:02.223744    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p" event={"ID":"37e161ce-b65e-42a3-ab4a-a669bb4a3880","Type":"ContainerStarted","Data":"fbfd029e8066a819ce7f8d61ad86a437d4d26447c53312674a66ea40f05ee772"}
Jul 28 20:14:03.854138 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:03.854098    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"]
Jul 28 20:14:03.854602 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:03.854331    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk" podUID="17d08bc1-7c6f-43d9-887c-e904c0561ce9" containerName="discovery" containerID="cri-o://f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e" gracePeriod=30
Jul 28 20:14:04.109888 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.109827    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:14:04.220680 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.220639    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-kubeconfig\") pod \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") "
Jul 28 20:14:04.220869 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.220759    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbtsb\" (UniqueName: \"kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-kube-api-access-xbtsb\") pod \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") "
Jul 28 20:14:04.220869 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.220806    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-token\") pod \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") "
Jul 28 20:14:04.220869 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.220838    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-ca-configmap\") pod \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") "
Jul 28 20:14:04.221036 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.220879    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d08bc1-7c6f-43d9-887c-e904c0561ce9-local-certs\") pod \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") "
Jul 28 20:14:04.221036 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.220910    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-dns-cert\") pod \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") "
Jul 28 20:14:04.221036 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.220944    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-cacerts\") pod \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\" (UID: \"17d08bc1-7c6f-43d9-887c-e904c0561ce9\") "
Jul 28 20:14:04.221386 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.221333    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "17d08bc1-7c6f-43d9-887c-e904c0561ce9" (UID: "17d08bc1-7c6f-43d9-887c-e904c0561ce9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:14:04.223439 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.223410    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d08bc1-7c6f-43d9-887c-e904c0561ce9-local-certs" (OuterVolumeSpecName: "local-certs") pod "17d08bc1-7c6f-43d9-887c-e904c0561ce9" (UID: "17d08bc1-7c6f-43d9-887c-e904c0561ce9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:14:04.223682 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.223633    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-kube-api-access-xbtsb" (OuterVolumeSpecName: "kube-api-access-xbtsb") pod "17d08bc1-7c6f-43d9-887c-e904c0561ce9" (UID: "17d08bc1-7c6f-43d9-887c-e904c0561ce9"). InnerVolumeSpecName "kube-api-access-xbtsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:14:04.223682 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.223632    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "17d08bc1-7c6f-43d9-887c-e904c0561ce9" (UID: "17d08bc1-7c6f-43d9-887c-e904c0561ce9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:14:04.223682 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.223660    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "17d08bc1-7c6f-43d9-887c-e904c0561ce9" (UID: "17d08bc1-7c6f-43d9-887c-e904c0561ce9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:14:04.223682 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.223643    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-token" (OuterVolumeSpecName: "istio-token") pod "17d08bc1-7c6f-43d9-887c-e904c0561ce9" (UID: "17d08bc1-7c6f-43d9-887c-e904c0561ce9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:14:04.223682 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.223643    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-cacerts" (OuterVolumeSpecName: "cacerts") pod "17d08bc1-7c6f-43d9-887c-e904c0561ce9" (UID: "17d08bc1-7c6f-43d9-887c-e904c0561ce9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:14:04.236484 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.236454    2575 generic.go:356] "Generic (PLEG): container finished" podID="17d08bc1-7c6f-43d9-887c-e904c0561ce9" containerID="f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e" exitCode=0
Jul 28 20:14:04.236598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.236497    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk" event={"ID":"17d08bc1-7c6f-43d9-887c-e904c0561ce9","Type":"ContainerDied","Data":"f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e"}
Jul 28 20:14:04.236598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.236525    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"
Jul 28 20:14:04.236598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.236578    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk" event={"ID":"17d08bc1-7c6f-43d9-887c-e904c0561ce9","Type":"ContainerDied","Data":"d7b718843fb84cbad00814113227cc25769c9418029f1c5578bc1d2b8022f849"}
Jul 28 20:14:04.236754 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.236607    2575 scope.go:117] "RemoveContainer" containerID="f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e"
Jul 28 20:14:04.247910 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.247889    2575 scope.go:117] "RemoveContainer" containerID="f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e"
Jul 28 20:14:04.248307 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:14:04.248261    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e\": container with ID starting with f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e not found: ID does not exist" containerID="f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e"
Jul 28 20:14:04.248421 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.248300    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e"} err="failed to get container status \"f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e\": rpc error: code = NotFound desc = could not find container \"f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e\": container with ID starting with f35a321a8ccb6fa610332538c5821ae0cab002a915ccf79ef353dec74db3d48e not found: ID does not exist"
Jul 28 20:14:04.275344 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.275315    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"]
Jul 28 20:14:04.282486 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.282465    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7pdmk"]
Jul 28 20:14:04.322084 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.322059    2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-kubeconfig\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:14:04.322084 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.322082    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbtsb\" (UniqueName: \"kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-kube-api-access-xbtsb\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:14:04.322248 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.322093    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-token\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:14:04.322248 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.322102    2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-ca-configmap\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:14:04.322248 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.322110    2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d08bc1-7c6f-43d9-887c-e904c0561ce9-local-certs\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:14:04.322248 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.322118    2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-istio-csr-dns-cert\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:14:04.322248 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.322126    2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17d08bc1-7c6f-43d9-887c-e904c0561ce9-cacerts\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:14:04.837367 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:04.837329    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17d08bc1-7c6f-43d9-887c-e904c0561ce9" path="/var/lib/kubelet/pods/17d08bc1-7c6f-43d9-887c-e904c0561ce9/volumes"
Jul 28 20:14:06.248216 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:06.248180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj" event={"ID":"3c3fde1b-a971-4276-9a68-a304440d6094","Type":"ContainerStarted","Data":"5f8af8787938c25d2a1c4bbb64a74b94e0a8c673487e2bc48ad16f52a5f89b2a"}
Jul 28 20:14:06.248706 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:06.248304    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:06.249877 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:06.249845    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p" event={"ID":"37e161ce-b65e-42a3-ab4a-a669bb4a3880","Type":"ContainerStarted","Data":"9cb80588ec9d3558be3dace1cec63206f59f69a6cadab6f81d12d2b8a4b39363"}
Jul 28 20:14:06.250078 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:06.249978    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"
Jul 28 20:14:06.267886 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:06.267839    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj" podStartSLOduration=1.869290825 podStartE2EDuration="5.267821843s" totalImagesPullingTime="3.398531018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:06.266159384 +0000 UTC m=+345.880396517" watchObservedRunningTime="2026-07-28 20:14:06.267821843 +0000 UTC m=+345.882058993"
Jul 28 20:14:06.285154 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:06.285102    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p" podStartSLOduration=1.323438285 podStartE2EDuration="5.285085924s" totalImagesPullingTime="3.961647639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:06.284576716 +0000 UTC m=+345.898813849" watchObservedRunningTime="2026-07-28 20:14:06.285085924 +0000 UTC m=+345.899323056"
Jul 28 20:14:07.202980 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:07.202952    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-g7mnm"
Jul 28 20:14:07.204899 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:07.204883    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-lb5h7"
Jul 28 20:14:25.045991 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:25.045951    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:25.149308 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:25.149277    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:37.257848 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:37.257816    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h677p"
Jul 28 20:14:37.261151 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:37.261129    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cqcqj"
Jul 28 20:14:50.001669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.001618    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:50.097874 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.097838    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:50.135697 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.135659    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6sggv"]
Jul 28 20:14:50.135955 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.135943    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17d08bc1-7c6f-43d9-887c-e904c0561ce9" containerName="discovery"
Jul 28 20:14:50.136008 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.135957    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="17d08bc1-7c6f-43d9-887c-e904c0561ce9" containerName="discovery"
Jul 28 20:14:50.136042 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.136017    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="17d08bc1-7c6f-43d9-887c-e904c0561ce9" containerName="discovery"
Jul 28 20:14:50.138069 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.138042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6sggv"
Jul 28 20:14:50.141339 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.141319    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 20:14:50.142570 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.142420    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-zm59r\""
Jul 28 20:14:50.142570 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.142464    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 20:14:50.157756 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.157734    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6sggv"]
Jul 28 20:14:50.189523 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.189486    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxvp\" (UniqueName: \"kubernetes.io/projected/30665728-0f06-49ea-90b9-51c45697ddec-kube-api-access-6hxvp\") pod \"kuadrant-operator-catalog-6sggv\" (UID: \"30665728-0f06-49ea-90b9-51c45697ddec\") " pod="kuadrant-system/kuadrant-operator-catalog-6sggv"
Jul 28 20:14:50.194938 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.194913    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:50.290113 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.290031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxvp\" (UniqueName: \"kubernetes.io/projected/30665728-0f06-49ea-90b9-51c45697ddec-kube-api-access-6hxvp\") pod \"kuadrant-operator-catalog-6sggv\" (UID: \"30665728-0f06-49ea-90b9-51c45697ddec\") " pod="kuadrant-system/kuadrant-operator-catalog-6sggv"
Jul 28 20:14:50.298282 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.298249    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxvp\" (UniqueName: \"kubernetes.io/projected/30665728-0f06-49ea-90b9-51c45697ddec-kube-api-access-6hxvp\") pod \"kuadrant-operator-catalog-6sggv\" (UID: \"30665728-0f06-49ea-90b9-51c45697ddec\") " pod="kuadrant-system/kuadrant-operator-catalog-6sggv"
Jul 28 20:14:50.448878 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.448846    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6sggv"
Jul 28 20:14:50.510376 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.508814    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6sggv"]
Jul 28 20:14:50.592570 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.592494    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6sggv"]
Jul 28 20:14:50.595822 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:14:50.595797    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30665728_0f06_49ea_90b9_51c45697ddec.slice/crio-d3c0fdf4177c2ac56b3ba3a575354b106b80f6c7eb6b429ff29320062150dab6 WatchSource:0}: Error finding container d3c0fdf4177c2ac56b3ba3a575354b106b80f6c7eb6b429ff29320062150dab6: Status 404 returned error can't find the container with id d3c0fdf4177c2ac56b3ba3a575354b106b80f6c7eb6b429ff29320062150dab6
Jul 28 20:14:50.710028 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.709994    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2pwvq"]
Jul 28 20:14:50.713076 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.713058    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2pwvq"
Jul 28 20:14:50.723425 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.723394    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2pwvq"]
Jul 28 20:14:50.794625 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.794572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l79w\" (UniqueName: \"kubernetes.io/projected/a0f6de5b-f022-4d5e-af24-25c3b0fca65d-kube-api-access-7l79w\") pod \"kuadrant-operator-catalog-2pwvq\" (UID: \"a0f6de5b-f022-4d5e-af24-25c3b0fca65d\") " pod="kuadrant-system/kuadrant-operator-catalog-2pwvq"
Jul 28 20:14:50.895583 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.895560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l79w\" (UniqueName: \"kubernetes.io/projected/a0f6de5b-f022-4d5e-af24-25c3b0fca65d-kube-api-access-7l79w\") pod \"kuadrant-operator-catalog-2pwvq\" (UID: \"a0f6de5b-f022-4d5e-af24-25c3b0fca65d\") " pod="kuadrant-system/kuadrant-operator-catalog-2pwvq"
Jul 28 20:14:50.904744 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:50.904719    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l79w\" (UniqueName: \"kubernetes.io/projected/a0f6de5b-f022-4d5e-af24-25c3b0fca65d-kube-api-access-7l79w\") pod \"kuadrant-operator-catalog-2pwvq\" (UID: \"a0f6de5b-f022-4d5e-af24-25c3b0fca65d\") " pod="kuadrant-system/kuadrant-operator-catalog-2pwvq"
Jul 28 20:14:51.022724 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:51.022683    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2pwvq"
Jul 28 20:14:51.163816 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:51.163788    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2pwvq"]
Jul 28 20:14:51.166058 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:14:51.166031    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f6de5b_f022_4d5e_af24_25c3b0fca65d.slice/crio-7f1c989ad2b4f17455103555c12dc36cdb6408db0930abcef508e1ffb46acc6f WatchSource:0}: Error finding container 7f1c989ad2b4f17455103555c12dc36cdb6408db0930abcef508e1ffb46acc6f: Status 404 returned error can't find the container with id 7f1c989ad2b4f17455103555c12dc36cdb6408db0930abcef508e1ffb46acc6f
Jul 28 20:14:51.431933 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:51.431850    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2pwvq" event={"ID":"a0f6de5b-f022-4d5e-af24-25c3b0fca65d","Type":"ContainerStarted","Data":"7f1c989ad2b4f17455103555c12dc36cdb6408db0930abcef508e1ffb46acc6f"}
Jul 28 20:14:51.433035 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:51.433006    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6sggv" event={"ID":"30665728-0f06-49ea-90b9-51c45697ddec","Type":"ContainerStarted","Data":"d3c0fdf4177c2ac56b3ba3a575354b106b80f6c7eb6b429ff29320062150dab6"}
Jul 28 20:14:53.443702 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:53.443657    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6sggv" event={"ID":"30665728-0f06-49ea-90b9-51c45697ddec","Type":"ContainerStarted","Data":"e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1"}
Jul 28 20:14:53.444193 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:53.443672    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-6sggv" podUID="30665728-0f06-49ea-90b9-51c45697ddec" containerName="registry-server" containerID="cri-o://e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1" gracePeriod=2
Jul 28 20:14:53.445063 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:53.445037    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2pwvq" event={"ID":"a0f6de5b-f022-4d5e-af24-25c3b0fca65d","Type":"ContainerStarted","Data":"6bd2928dd75bd50441ed832ac3eb973a50254da2b00960d1af6a1fe29a3f5aee"}
Jul 28 20:14:53.463373 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:53.463326    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6sggv" podStartSLOduration=1.476494562 podStartE2EDuration="3.463310728s" totalImagesPullingTime="1.986816166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:53.46201459 +0000 UTC m=+393.076251723" watchObservedRunningTime="2026-07-28 20:14:53.463310728 +0000 UTC m=+393.077547855"
Jul 28 20:14:53.483822 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:53.483770    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2pwvq" podStartSLOduration=2.0665139 podStartE2EDuration="3.483754399s" totalImagesPullingTime="1.417240499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:53.482267768 +0000 UTC m=+393.096504907" watchObservedRunningTime="2026-07-28 20:14:53.483754399 +0000 UTC m=+393.097991532"
Jul 28 20:14:53.683125 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:53.683098    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6sggv"
Jul 28 20:14:53.718496 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:53.718400    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hxvp\" (UniqueName: \"kubernetes.io/projected/30665728-0f06-49ea-90b9-51c45697ddec-kube-api-access-6hxvp\") pod \"30665728-0f06-49ea-90b9-51c45697ddec\" (UID: \"30665728-0f06-49ea-90b9-51c45697ddec\") "
Jul 28 20:14:53.720586 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:53.720554    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30665728-0f06-49ea-90b9-51c45697ddec-kube-api-access-6hxvp" (OuterVolumeSpecName: "kube-api-access-6hxvp") pod "30665728-0f06-49ea-90b9-51c45697ddec" (UID: "30665728-0f06-49ea-90b9-51c45697ddec"). InnerVolumeSpecName "kube-api-access-6hxvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:14:53.818931 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:53.818899    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxvp\" (UniqueName: \"kubernetes.io/projected/30665728-0f06-49ea-90b9-51c45697ddec-kube-api-access-6hxvp\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:14:54.449796 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:54.449759    2575 generic.go:356] "Generic (PLEG): container finished" podID="30665728-0f06-49ea-90b9-51c45697ddec" containerID="e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1" exitCode=0
Jul 28 20:14:54.450192 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:54.449801    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6sggv"
Jul 28 20:14:54.450192 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:54.449847    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6sggv" event={"ID":"30665728-0f06-49ea-90b9-51c45697ddec","Type":"ContainerDied","Data":"e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1"}
Jul 28 20:14:54.450192 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:54.449891    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6sggv" event={"ID":"30665728-0f06-49ea-90b9-51c45697ddec","Type":"ContainerDied","Data":"d3c0fdf4177c2ac56b3ba3a575354b106b80f6c7eb6b429ff29320062150dab6"}
Jul 28 20:14:54.450192 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:54.449912    2575 scope.go:117] "RemoveContainer" containerID="e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1"
Jul 28 20:14:54.458593 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:54.458574    2575 scope.go:117] "RemoveContainer" containerID="e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1"
Jul 28 20:14:54.458839 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:14:54.458821    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1\": container with ID starting with e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1 not found: ID does not exist" containerID="e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1"
Jul 28 20:14:54.458907 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:54.458847    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1"} err="failed to get container status \"e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1\": rpc error: code = NotFound desc = could not find container \"e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1\": container with ID starting with e24db9d9de31d23a9b1be62a7576310dff0fe19e51c54281b68be4306574e3a1 not found: ID does not exist"
Jul 28 20:14:54.469656 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:54.469632    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6sggv"]
Jul 28 20:14:54.473563 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:54.473543    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6sggv"]
Jul 28 20:14:54.834512 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:14:54.834440    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30665728-0f06-49ea-90b9-51c45697ddec" path="/var/lib/kubelet/pods/30665728-0f06-49ea-90b9-51c45697ddec/volumes"
Jul 28 20:15:01.023369 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:01.023274    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-2pwvq"
Jul 28 20:15:01.023369 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:01.023330    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-2pwvq"
Jul 28 20:15:01.045248 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:01.045204    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-2pwvq"
Jul 28 20:15:01.500017 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:01.499990    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-2pwvq"
Jul 28 20:15:17.055565 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:17.055529    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:17.115666 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:17.115634    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:22.294712 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:22.294677    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:24.706138 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.706104    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv"]
Jul 28 20:15:24.706524 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.706452    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30665728-0f06-49ea-90b9-51c45697ddec" containerName="registry-server"
Jul 28 20:15:24.706524 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.706468    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="30665728-0f06-49ea-90b9-51c45697ddec" containerName="registry-server"
Jul 28 20:15:24.706594 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.706543    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="30665728-0f06-49ea-90b9-51c45697ddec" containerName="registry-server"
Jul 28 20:15:24.713983 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.713961    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv"
Jul 28 20:15:24.716362 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.716341    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zst2s\""
Jul 28 20:15:24.716901 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.716884    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 28 20:15:24.719600 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.719575    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv"]
Jul 28 20:15:24.865389 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.865356    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flntd\" (UniqueName: \"kubernetes.io/projected/44b7304b-3f3d-479b-a6f6-1923e2070a0b-kube-api-access-flntd\") pod \"dns-operator-controller-manager-864dd78c4c-jz6wv\" (UID: \"44b7304b-3f3d-479b-a6f6-1923e2070a0b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv"
Jul 28 20:15:24.966580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.966496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flntd\" (UniqueName: \"kubernetes.io/projected/44b7304b-3f3d-479b-a6f6-1923e2070a0b-kube-api-access-flntd\") pod \"dns-operator-controller-manager-864dd78c4c-jz6wv\" (UID: \"44b7304b-3f3d-479b-a6f6-1923e2070a0b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv"
Jul 28 20:15:24.974500 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:24.974471    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flntd\" (UniqueName: \"kubernetes.io/projected/44b7304b-3f3d-479b-a6f6-1923e2070a0b-kube-api-access-flntd\") pod \"dns-operator-controller-manager-864dd78c4c-jz6wv\" (UID: \"44b7304b-3f3d-479b-a6f6-1923e2070a0b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv"
Jul 28 20:15:25.024176 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:25.024139    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv"
Jul 28 20:15:25.155662 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:25.155626    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv"]
Jul 28 20:15:25.160563 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:15:25.160536    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b7304b_3f3d_479b_a6f6_1923e2070a0b.slice/crio-54300bb9789c2071a4b923c5a1712f06c18d318eef54b3373dac7d5d3aa4f753 WatchSource:0}: Error finding container 54300bb9789c2071a4b923c5a1712f06c18d318eef54b3373dac7d5d3aa4f753: Status 404 returned error can't find the container with id 54300bb9789c2071a4b923c5a1712f06c18d318eef54b3373dac7d5d3aa4f753
Jul 28 20:15:25.577159 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:25.577125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv" event={"ID":"44b7304b-3f3d-479b-a6f6-1923e2070a0b","Type":"ContainerStarted","Data":"54300bb9789c2071a4b923c5a1712f06c18d318eef54b3373dac7d5d3aa4f753"}
Jul 28 20:15:26.369432 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:26.369399    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:27.512630 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:27.512533    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:27.587540 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:27.587505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv" event={"ID":"44b7304b-3f3d-479b-a6f6-1923e2070a0b","Type":"ContainerStarted","Data":"ad37a3dcb39525a13e2f1158fa341f3754e1cdd855c694d63805c738a7db7944"}
Jul 28 20:15:27.587703 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:27.587642    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv"
Jul 28 20:15:27.604964 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:27.604920    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv" podStartSLOduration=1.580122072 podStartE2EDuration="3.604905519s" totalImagesPullingTime="2.024783447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:27.603291136 +0000 UTC m=+427.217528280" watchObservedRunningTime="2026-07-28 20:15:27.604905519 +0000 UTC m=+427.219142652"
Jul 28 20:15:27.608966 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:27.608934    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:27.664379 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:27.664351    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:28.181564 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:28.181531    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"]
Jul 28 20:15:28.184765 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:28.184747    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:28.186964 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:28.186946    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tx5qg\""
Jul 28 20:15:28.192918 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:28.192896    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"]
Jul 28 20:15:28.293900 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:28.293861    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlv89\" (UniqueName: \"kubernetes.io/projected/7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2-kube-api-access-xlv89\") pod \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" (UID: \"7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:28.395312 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:28.395277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlv89\" (UniqueName: \"kubernetes.io/projected/7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2-kube-api-access-xlv89\") pod \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" (UID: \"7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:28.411737 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:28.411709    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlv89\" (UniqueName: \"kubernetes.io/projected/7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2-kube-api-access-xlv89\") pod \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" (UID: \"7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:28.495405 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:28.495319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:28.625015 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:28.624979    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"]
Jul 28 20:15:28.627493 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:15:28.627464    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a75667a_ec4f_48fd_b7e7_507ea7e8c5e2.slice/crio-d115176c8c232d0a7d84c4c519679a238ee49abb770fe4ea84fbc215e12ed043 WatchSource:0}: Error finding container d115176c8c232d0a7d84c4c519679a238ee49abb770fe4ea84fbc215e12ed043: Status 404 returned error can't find the container with id d115176c8c232d0a7d84c4c519679a238ee49abb770fe4ea84fbc215e12ed043
Jul 28 20:15:28.730038 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:28.730000    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:29.119374 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:29.119338    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:29.288926 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:29.288893    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:29.601665 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:29.601630    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9" event={"ID":"7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2","Type":"ContainerStarted","Data":"d115176c8c232d0a7d84c4c519679a238ee49abb770fe4ea84fbc215e12ed043"}
Jul 28 20:15:31.507589 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:31.507557    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:31.605016 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:31.604966    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:31.610439 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:31.610412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9" event={"ID":"7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2","Type":"ContainerStarted","Data":"348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7"}
Jul 28 20:15:31.610572 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:31.610487    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:31.629263 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:31.629200    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9" podStartSLOduration=1.679411381 podStartE2EDuration="3.629186351s" totalImagesPullingTime="1.94977497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:31.627757623 +0000 UTC m=+431.241994755" watchObservedRunningTime="2026-07-28 20:15:31.629186351 +0000 UTC m=+431.243423484"
Jul 28 20:15:31.884140 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:31.884059    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b"]
Jul 28 20:15:31.887346 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:31.887327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b"
Jul 28 20:15:31.889625 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:31.889603    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-m6r26\""
Jul 28 20:15:31.896592 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:31.896562    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b"]
Jul 28 20:15:32.026364 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:32.026323    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lk4\" (UniqueName: \"kubernetes.io/projected/44fd7dfb-b62c-417d-9505-29daa9d843a3-kube-api-access-64lk4\") pod \"authorino-operator-75cc8d5cb7-p7d7b\" (UID: \"44fd7dfb-b62c-417d-9505-29daa9d843a3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b"
Jul 28 20:15:32.127340 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:32.127303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lk4\" (UniqueName: \"kubernetes.io/projected/44fd7dfb-b62c-417d-9505-29daa9d843a3-kube-api-access-64lk4\") pod \"authorino-operator-75cc8d5cb7-p7d7b\" (UID: \"44fd7dfb-b62c-417d-9505-29daa9d843a3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b"
Jul 28 20:15:32.138038 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:32.137986    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lk4\" (UniqueName: \"kubernetes.io/projected/44fd7dfb-b62c-417d-9505-29daa9d843a3-kube-api-access-64lk4\") pod \"authorino-operator-75cc8d5cb7-p7d7b\" (UID: \"44fd7dfb-b62c-417d-9505-29daa9d843a3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b"
Jul 28 20:15:32.198157 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:32.198111    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b"
Jul 28 20:15:32.340658 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:32.340630    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b"]
Jul 28 20:15:32.343006 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:15:32.342982    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fd7dfb_b62c_417d_9505_29daa9d843a3.slice/crio-8a6e12c91e5408b196676a3b4bc07d4ab221ff9a8271aa77e67a87332acc6d4f WatchSource:0}: Error finding container 8a6e12c91e5408b196676a3b4bc07d4ab221ff9a8271aa77e67a87332acc6d4f: Status 404 returned error can't find the container with id 8a6e12c91e5408b196676a3b4bc07d4ab221ff9a8271aa77e67a87332acc6d4f
Jul 28 20:15:32.616158 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:32.616125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b" event={"ID":"44fd7dfb-b62c-417d-9505-29daa9d843a3","Type":"ContainerStarted","Data":"8a6e12c91e5408b196676a3b4bc07d4ab221ff9a8271aa77e67a87332acc6d4f"}
Jul 28 20:15:32.677318 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:32.677284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:34.337112 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:34.337070    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:35.631738 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:35.631707    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b" event={"ID":"44fd7dfb-b62c-417d-9505-29daa9d843a3","Type":"ContainerStarted","Data":"fb077444f28ee9987ae64a83b45a9ceed6ca80282ffe1ff47d605ad199c2a5db"}
Jul 28 20:15:35.632551 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:35.631838    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b"
Jul 28 20:15:35.647156 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:35.647104    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b" podStartSLOduration=2.197564156 podStartE2EDuration="4.647087794s" totalImagesPullingTime="2.449523638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:35.64624626 +0000 UTC m=+435.260483388" watchObservedRunningTime="2026-07-28 20:15:35.647087794 +0000 UTC m=+435.261324930"
Jul 28 20:15:36.323321 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:36.323284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:36.889126 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:36.889088    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"]
Jul 28 20:15:36.892455 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:36.892435    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:36.894774 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:36.894747    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4qq2g\""
Jul 28 20:15:36.906392 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:36.906359    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"]
Jul 28 20:15:36.969733 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:36.969696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9883e6ee-c792-4df0-a60e-b98478ff497e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" (UID: \"9883e6ee-c792-4df0-a60e-b98478ff497e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:36.969923 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:36.969765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-999rx\" (UniqueName: \"kubernetes.io/projected/9883e6ee-c792-4df0-a60e-b98478ff497e-kube-api-access-999rx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" (UID: \"9883e6ee-c792-4df0-a60e-b98478ff497e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:37.070696 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:37.070655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9883e6ee-c792-4df0-a60e-b98478ff497e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" (UID: \"9883e6ee-c792-4df0-a60e-b98478ff497e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:37.070853 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:37.070714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999rx\" (UniqueName: \"kubernetes.io/projected/9883e6ee-c792-4df0-a60e-b98478ff497e-kube-api-access-999rx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" (UID: \"9883e6ee-c792-4df0-a60e-b98478ff497e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:37.071127 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:37.071102    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9883e6ee-c792-4df0-a60e-b98478ff497e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" (UID: \"9883e6ee-c792-4df0-a60e-b98478ff497e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:37.079727 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:37.079698    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-999rx\" (UniqueName: \"kubernetes.io/projected/9883e6ee-c792-4df0-a60e-b98478ff497e-kube-api-access-999rx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" (UID: \"9883e6ee-c792-4df0-a60e-b98478ff497e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:37.201901 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:37.201867    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:37.345885 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:37.345858    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"]
Jul 28 20:15:37.348529 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:15:37.348505    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9883e6ee_c792_4df0_a60e_b98478ff497e.slice/crio-eac836660ce36f19ebae33be09346abca64813082c5a41c9bc75be086cdd1cfa WatchSource:0}: Error finding container eac836660ce36f19ebae33be09346abca64813082c5a41c9bc75be086cdd1cfa: Status 404 returned error can't find the container with id eac836660ce36f19ebae33be09346abca64813082c5a41c9bc75be086cdd1cfa
Jul 28 20:15:37.641302 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:37.641198    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p" event={"ID":"9883e6ee-c792-4df0-a60e-b98478ff497e","Type":"ContainerStarted","Data":"eac836660ce36f19ebae33be09346abca64813082c5a41c9bc75be086cdd1cfa"}
Jul 28 20:15:38.299902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:38.299861    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:38.391432 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:38.391398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:38.596316 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:38.596215    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jz6wv"
Jul 28 20:15:41.689836 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:41.689791    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:41.788783 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:41.788743    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:42.619095 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:42.618963    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:44.680734 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:44.680694    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p" event={"ID":"9883e6ee-c792-4df0-a60e-b98478ff497e","Type":"ContainerStarted","Data":"4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f"}
Jul 28 20:15:44.681138 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:44.680825    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:44.700586 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:44.700537    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p" podStartSLOduration=2.408375863 podStartE2EDuration="8.700523151s" totalImagesPullingTime="6.292147288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:44.698254942 +0000 UTC m=+444.312492073" watchObservedRunningTime="2026-07-28 20:15:44.700523151 +0000 UTC m=+444.314760284"
Jul 28 20:15:46.267727 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:46.267693    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:46.327826 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:46.327792    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:46.473213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:46.473172    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:46.536190 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:46.536112    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:46.610302 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:46.610269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:46.638342 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:46.638318    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p7d7b"
Jul 28 20:15:48.539400 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:48.539371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:50.223967 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:50.223931    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:50.323768 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:50.323709    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:55.686541 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:55.686510    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:56.662908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.662875    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"]
Jul 28 20:15:56.669840 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.669817    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:56.684870 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.684844    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"]
Jul 28 20:15:56.728620 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.728585    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nzmm\" (UniqueName: \"kubernetes.io/projected/ac0d5310-3733-4c7f-a8e1-87175a8e052a-kube-api-access-8nzmm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" (UID: \"ac0d5310-3733-4c7f-a8e1-87175a8e052a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:56.729019 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.728630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac0d5310-3733-4c7f-a8e1-87175a8e052a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" (UID: \"ac0d5310-3733-4c7f-a8e1-87175a8e052a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:56.829549 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.829508    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzmm\" (UniqueName: \"kubernetes.io/projected/ac0d5310-3733-4c7f-a8e1-87175a8e052a-kube-api-access-8nzmm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" (UID: \"ac0d5310-3733-4c7f-a8e1-87175a8e052a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:56.829549 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.829554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac0d5310-3733-4c7f-a8e1-87175a8e052a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" (UID: \"ac0d5310-3733-4c7f-a8e1-87175a8e052a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:56.829986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.829963    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac0d5310-3733-4c7f-a8e1-87175a8e052a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" (UID: \"ac0d5310-3733-4c7f-a8e1-87175a8e052a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:56.840808 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.840782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzmm\" (UniqueName: \"kubernetes.io/projected/ac0d5310-3733-4c7f-a8e1-87175a8e052a-kube-api-access-8nzmm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" (UID: \"ac0d5310-3733-4c7f-a8e1-87175a8e052a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:56.866169 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.866140    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:56.966556 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.966478    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:56.982807 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:56.982539    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:57.117683 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.117598    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"]
Jul 28 20:15:57.120612 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:15:57.120575    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0d5310_3733_4c7f_a8e1_87175a8e052a.slice/crio-9f0face149c38c8d4aa787b03a850a865b983d03cb344844ea8f39f79007ce99 WatchSource:0}: Error finding container 9f0face149c38c8d4aa787b03a850a865b983d03cb344844ea8f39f79007ce99: Status 404 returned error can't find the container with id 9f0face149c38c8d4aa787b03a850a865b983d03cb344844ea8f39f79007ce99
Jul 28 20:15:57.388471 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.388387    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"]
Jul 28 20:15:57.392025 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.391999    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"]
Jul 28 20:15:57.404838 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.404815    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"]
Jul 28 20:15:57.405053 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.405011    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" containerName="manager" containerID="cri-o://4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f" gracePeriod=2
Jul 28 20:15:57.415072 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.415049    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"]
Jul 28 20:15:57.425180 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.425139    2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9883e6ee-c792-4df0-a60e-b98478ff497e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-57n2p\": pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.427560 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.427532    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"]
Jul 28 20:15:57.427947 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.427762    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" containerName="manager" containerID="cri-o://348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7" gracePeriod=2
Jul 28 20:15:57.435515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.435490    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"]
Jul 28 20:15:57.435928 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.435912    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" containerName="manager"
Jul 28 20:15:57.435996 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.435930    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" containerName="manager"
Jul 28 20:15:57.435996 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.435961    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" containerName="manager"
Jul 28 20:15:57.435996 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.435969    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" containerName="manager"
Jul 28 20:15:57.436106 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.436052    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" containerName="manager"
Jul 28 20:15:57.436106 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.436069    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" containerName="manager"
Jul 28 20:15:57.439749 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.439729    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:15:57.445381 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.445355    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"]
Jul 28 20:15:57.447906 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.447560    2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-4rzq9\": pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.449578 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.449543    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.450159 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.450138    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp"]
Jul 28 20:15:57.450649 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.450626    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" containerName="manager"
Jul 28 20:15:57.450785 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.450651    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" containerName="manager"
Jul 28 20:15:57.450785 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.450734    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" containerName="manager"
Jul 28 20:15:57.454196 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.454174    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"]
Jul 28 20:15:57.454300 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.454285    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp"
Jul 28 20:15:57.462389 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.462368    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp"]
Jul 28 20:15:57.473896 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.473861    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.475615 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.475582    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.477384 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.477350    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.498833 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.498803    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.536715 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.536688    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wrx\" (UniqueName: \"kubernetes.io/projected/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-kube-api-access-q5wrx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9njr\" (UID: \"ab0bb0fe-4b29-4a32-b449-78ed1638b70b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:15:57.536851 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.536731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9njr\" (UID: \"ab0bb0fe-4b29-4a32-b449-78ed1638b70b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:15:57.536851 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.536757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxsm\" (UniqueName: \"kubernetes.io/projected/dda2a715-536e-4f18-bc21-402087b7960a-kube-api-access-7sxsm\") pod \"limitador-operator-controller-manager-859d7b78bd-6t2qp\" (UID: \"dda2a715-536e-4f18-bc21-402087b7960a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp"
Jul 28 20:15:57.637736 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.637696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wrx\" (UniqueName: \"kubernetes.io/projected/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-kube-api-access-q5wrx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9njr\" (UID: \"ab0bb0fe-4b29-4a32-b449-78ed1638b70b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:15:57.637923 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.637750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9njr\" (UID: \"ab0bb0fe-4b29-4a32-b449-78ed1638b70b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:15:57.637923 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.637775    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxsm\" (UniqueName: \"kubernetes.io/projected/dda2a715-536e-4f18-bc21-402087b7960a-kube-api-access-7sxsm\") pod \"limitador-operator-controller-manager-859d7b78bd-6t2qp\" (UID: \"dda2a715-536e-4f18-bc21-402087b7960a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp"
Jul 28 20:15:57.638245 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.638207    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9njr\" (UID: \"ab0bb0fe-4b29-4a32-b449-78ed1638b70b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:15:57.645394 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.645366    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wrx\" (UniqueName: \"kubernetes.io/projected/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-kube-api-access-q5wrx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9njr\" (UID: \"ab0bb0fe-4b29-4a32-b449-78ed1638b70b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:15:57.645537 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.645507    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxsm\" (UniqueName: \"kubernetes.io/projected/dda2a715-536e-4f18-bc21-402087b7960a-kube-api-access-7sxsm\") pod \"limitador-operator-controller-manager-859d7b78bd-6t2qp\" (UID: \"dda2a715-536e-4f18-bc21-402087b7960a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp"
Jul 28 20:15:57.655621 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.655602    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.657609 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.657585    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.659305 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.659282    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.663700 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.663682    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.665630 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.665606    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.667137 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.667115    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.738134 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.738102    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlv89\" (UniqueName: \"kubernetes.io/projected/7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2-kube-api-access-xlv89\") pod \"7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2\" (UID: \"7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2\") "
Jul 28 20:15:57.738599 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.738183    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9883e6ee-c792-4df0-a60e-b98478ff497e-extensions-socket-volume\") pod \"9883e6ee-c792-4df0-a60e-b98478ff497e\" (UID: \"9883e6ee-c792-4df0-a60e-b98478ff497e\") "
Jul 28 20:15:57.738599 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.738220    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-999rx\" (UniqueName: \"kubernetes.io/projected/9883e6ee-c792-4df0-a60e-b98478ff497e-kube-api-access-999rx\") pod \"9883e6ee-c792-4df0-a60e-b98478ff497e\" (UID: \"9883e6ee-c792-4df0-a60e-b98478ff497e\") "
Jul 28 20:15:57.738735 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.738711    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9883e6ee-c792-4df0-a60e-b98478ff497e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9883e6ee-c792-4df0-a60e-b98478ff497e" (UID: "9883e6ee-c792-4df0-a60e-b98478ff497e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:15:57.739303 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.739271    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" containerID="348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7" exitCode=0
Jul 28 20:15:57.739418 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.739329    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.739418 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.739362    2575 scope.go:117] "RemoveContainer" containerID="348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7"
Jul 28 20:15:57.740689 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.740660    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2-kube-api-access-xlv89" (OuterVolumeSpecName: "kube-api-access-xlv89") pod "7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" (UID: "7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2"). InnerVolumeSpecName "kube-api-access-xlv89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:15:57.740819 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.740795    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9883e6ee-c792-4df0-a60e-b98478ff497e-kube-api-access-999rx" (OuterVolumeSpecName: "kube-api-access-999rx") pod "9883e6ee-c792-4df0-a60e-b98478ff497e" (UID: "9883e6ee-c792-4df0-a60e-b98478ff497e"). InnerVolumeSpecName "kube-api-access-999rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:15:57.741106 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.741079    2575 generic.go:356] "Generic (PLEG): container finished" podID="9883e6ee-c792-4df0-a60e-b98478ff497e" containerID="4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f" exitCode=0
Jul 28 20:15:57.741196 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.741146    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.741286 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.741197    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.742948 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.742926    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.743241 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.743213    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt" container="manager"
Jul 28 20:15:57.744663 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.744632    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.746319 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.746294    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.747994 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.747963    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:57.749718 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.749690    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.750099 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.750078    2575 scope.go:117] "RemoveContainer" containerID="348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7"
Jul 28 20:15:57.750418 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.750397    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7\": container with ID starting with 348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7 not found: ID does not exist" containerID="348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7"
Jul 28 20:15:57.750480 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.750428    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7"} err="failed to get container status \"348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7\": rpc error: code = NotFound desc = could not find container \"348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7\": container with ID starting with 348bac1390563015069e0c9dec9fb3827bad684e9105ee2464f58f62836f07a7 not found: ID does not exist"
Jul 28 20:15:57.750480 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.750444    2575 scope.go:117] "RemoveContainer" containerID="4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f"
Jul 28 20:15:57.751357 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.751334    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.755246 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.755186    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:57.756757 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.756728    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:57.758346 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.758319    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:57.759671 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.759656    2575 scope.go:117] "RemoveContainer" containerID="4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f"
Jul 28 20:15:57.759905 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:57.759885    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f\": container with ID starting with 4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f not found: ID does not exist" containerID="4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f"
Jul 28 20:15:57.759958 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.759913    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f"} err="failed to get container status \"4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f\": rpc error: code = NotFound desc = could not find container \"4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f\": container with ID starting with 4811d9a041b5d41a00cc465c34a794ba185047a2d992072b77b3a60658aea93f not found: ID does not exist"
Jul 28 20:15:57.825372 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.825335    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:15:57.831407 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.831385    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp"
Jul 28 20:15:57.838921 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.838895    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9883e6ee-c792-4df0-a60e-b98478ff497e-extensions-socket-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:15:57.839035 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.838925    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-999rx\" (UniqueName: \"kubernetes.io/projected/9883e6ee-c792-4df0-a60e-b98478ff497e-kube-api-access-999rx\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:15:57.839035 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.838936    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlv89\" (UniqueName: \"kubernetes.io/projected/7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2-kube-api-access-xlv89\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:15:57.985668 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:57.985623    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"]
Jul 28 20:15:57.990897 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:15:57.990864    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0bb0fe_4b29_4a32_b449_78ed1638b70b.slice/crio-f5afeab36d4170c0d03027cf3a8a168f65dee34b6b1b7b37caa00d24f3148602 WatchSource:0}: Error finding container f5afeab36d4170c0d03027cf3a8a168f65dee34b6b1b7b37caa00d24f3148602: Status 404 returned error can't find the container with id f5afeab36d4170c0d03027cf3a8a168f65dee34b6b1b7b37caa00d24f3148602
Jul 28 20:15:58.028801 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.028730    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp"]
Jul 28 20:15:58.057024 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:58.056992    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:58.058534 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:58.058509    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:58.060120 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:58.060094    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:58.749176 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.749144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp" event={"ID":"dda2a715-536e-4f18-bc21-402087b7960a","Type":"ContainerStarted","Data":"f74b82b8e683ba5f1d14e346c66b391836dc277ae0928c554f853e8109563d11"}
Jul 28 20:15:58.749663 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.749207    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp"
Jul 28 20:15:58.749663 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.749245    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp" event={"ID":"dda2a715-536e-4f18-bc21-402087b7960a","Type":"ContainerStarted","Data":"d8bce44e44d84f2af8824a22b5d28cb2ced534fe334652d84f72f599a02d4c4f"}
Jul 28 20:15:58.751205 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:58.751169    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:58.751687 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.751636    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" containerName="manager" containerID="cri-o://f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351" gracePeriod=2
Jul 28 20:15:58.752887 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:58.752860    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:58.753088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.753065    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr" event={"ID":"ab0bb0fe-4b29-4a32-b449-78ed1638b70b","Type":"ContainerStarted","Data":"27735a4cbaf8be20837302fcde88320e93e55b4d993bfe0a752914cdca2a68dc"}
Jul 28 20:15:58.753152 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.753112    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr" event={"ID":"ab0bb0fe-4b29-4a32-b449-78ed1638b70b","Type":"ContainerStarted","Data":"f5afeab36d4170c0d03027cf3a8a168f65dee34b6b1b7b37caa00d24f3148602"}
Jul 28 20:15:58.753152 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.753137    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:15:58.772426 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.772377    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp" podStartSLOduration=1.772359297 podStartE2EDuration="1.772359297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:58.772076224 +0000 UTC m=+458.386313382" watchObservedRunningTime="2026-07-28 20:15:58.772359297 +0000 UTC m=+458.386596431"
Jul 28 20:15:58.773933 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:58.773911    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:58.775544 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:58.775525    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-57n2p\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57n2p"
Jul 28 20:15:58.777259 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:58.777212    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:58.793924 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.793887    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr" podStartSLOduration=1.793875709 podStartE2EDuration="1.793875709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:58.792515079 +0000 UTC m=+458.406752222" watchObservedRunningTime="2026-07-28 20:15:58.793875709 +0000 UTC m=+458.408112841"
Jul 28 20:15:58.794195 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:58.794167    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4rzq9\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4rzq9"
Jul 28 20:15:58.837089 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.837049    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2" path="/var/lib/kubelet/pods/7a75667a-ec4f-48fd-b7e7-507ea7e8c5e2/volumes"
Jul 28 20:15:58.837489 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.837470    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9883e6ee-c792-4df0-a60e-b98478ff497e" path="/var/lib/kubelet/pods/9883e6ee-c792-4df0-a60e-b98478ff497e/volumes"
Jul 28 20:15:58.989434 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:58.989411    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:59.069766 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.069684    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nzmm\" (UniqueName: \"kubernetes.io/projected/ac0d5310-3733-4c7f-a8e1-87175a8e052a-kube-api-access-8nzmm\") pod \"ac0d5310-3733-4c7f-a8e1-87175a8e052a\" (UID: \"ac0d5310-3733-4c7f-a8e1-87175a8e052a\") "
Jul 28 20:15:59.069766 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.069734    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac0d5310-3733-4c7f-a8e1-87175a8e052a-extensions-socket-volume\") pod \"ac0d5310-3733-4c7f-a8e1-87175a8e052a\" (UID: \"ac0d5310-3733-4c7f-a8e1-87175a8e052a\") "
Jul 28 20:15:59.070034 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.070012    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0d5310-3733-4c7f-a8e1-87175a8e052a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ac0d5310-3733-4c7f-a8e1-87175a8e052a" (UID: "ac0d5310-3733-4c7f-a8e1-87175a8e052a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:15:59.071742 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.071713    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0d5310-3733-4c7f-a8e1-87175a8e052a-kube-api-access-8nzmm" (OuterVolumeSpecName: "kube-api-access-8nzmm") pod "ac0d5310-3733-4c7f-a8e1-87175a8e052a" (UID: "ac0d5310-3733-4c7f-a8e1-87175a8e052a"). InnerVolumeSpecName "kube-api-access-8nzmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:15:59.170900 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.170866    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nzmm\" (UniqueName: \"kubernetes.io/projected/ac0d5310-3733-4c7f-a8e1-87175a8e052a-kube-api-access-8nzmm\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:15:59.170900 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.170893    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac0d5310-3733-4c7f-a8e1-87175a8e052a-extensions-socket-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:15:59.760733 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.760697    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" containerID="f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351" exitCode=2
Jul 28 20:15:59.761216 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.760747    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:59.761216 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.760807    2575 scope.go:117] "RemoveContainer" containerID="f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351"
Jul 28 20:15:59.762881 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:59.762845    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:15:59.769943 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.769926    2575 scope.go:117] "RemoveContainer" containerID="f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351"
Jul 28 20:15:59.770207 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:59.770187    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351\": container with ID starting with f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351 not found: ID does not exist" containerID="f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351"
Jul 28 20:15:59.770285 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:15:59.770216    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351"} err="failed to get container status \"f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351\": rpc error: code = NotFound desc = could not find container \"f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351\": container with ID starting with f26011a18b3e1d98cc633558de1bd5eb91b48ed060e6421a0b21918e49e06351 not found: ID does not exist"
Jul 28 20:15:59.771410 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:15:59.771369    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:16:00.077774 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:00.077699    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:00.184634 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:00.184600    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:00.837668 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:16:00.837616    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-66rzt\" is forbidden: User \"system:node:ip-10-0-140-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-243.ec2.internal' and this object" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-66rzt"
Jul 28 20:16:00.840487 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:00.840460    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac0d5310-3733-4c7f-a8e1-87175a8e052a" path="/var/lib/kubelet/pods/ac0d5310-3733-4c7f-a8e1-87175a8e052a/volumes"
Jul 28 20:16:04.575974 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:04.575942    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:04.657891 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:04.657856    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:04.720057 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:04.720032    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:08.413217 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:08.413185    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:08.512348 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:08.512315    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:09.762821 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:09.762789    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t2qp"
Jul 28 20:16:09.763272 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:09.762863    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:16:15.696987 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:15.696948    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:15.752621 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:15.752591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:24.970642 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:24.970605    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:25.071207 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:25.071178    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:26.667472 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.667440    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"]
Jul 28 20:16:26.667872 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.667654    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr" podUID="ab0bb0fe-4b29-4a32-b449-78ed1638b70b" containerName="manager" containerID="cri-o://27735a4cbaf8be20837302fcde88320e93e55b4d993bfe0a752914cdca2a68dc" gracePeriod=10
Jul 28 20:16:26.881403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.881372    2575 generic.go:356] "Generic (PLEG): container finished" podID="ab0bb0fe-4b29-4a32-b449-78ed1638b70b" containerID="27735a4cbaf8be20837302fcde88320e93e55b4d993bfe0a752914cdca2a68dc" exitCode=0
Jul 28 20:16:26.881565 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.881452    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr" event={"ID":"ab0bb0fe-4b29-4a32-b449-78ed1638b70b","Type":"ContainerDied","Data":"27735a4cbaf8be20837302fcde88320e93e55b4d993bfe0a752914cdca2a68dc"}
Jul 28 20:16:26.901204 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.901177    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"]
Jul 28 20:16:26.904533 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.904515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:16:26.909104 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.909086    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:16:26.919306 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.919260    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"]
Jul 28 20:16:26.987736 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.987706    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5wrx\" (UniqueName: \"kubernetes.io/projected/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-kube-api-access-q5wrx\") pod \"ab0bb0fe-4b29-4a32-b449-78ed1638b70b\" (UID: \"ab0bb0fe-4b29-4a32-b449-78ed1638b70b\") "
Jul 28 20:16:26.987883 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.987764    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-extensions-socket-volume\") pod \"ab0bb0fe-4b29-4a32-b449-78ed1638b70b\" (UID: \"ab0bb0fe-4b29-4a32-b449-78ed1638b70b\") "
Jul 28 20:16:26.987883 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.987869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/373dceac-889f-4689-943c-c3a13893e917-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8np8h\" (UID: \"373dceac-889f-4689-943c-c3a13893e917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:16:26.988009 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.987988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxpvt\" (UniqueName: \"kubernetes.io/projected/373dceac-889f-4689-943c-c3a13893e917-kube-api-access-fxpvt\") pod \"kuadrant-operator-controller-manager-86878c769f-8np8h\" (UID: \"373dceac-889f-4689-943c-c3a13893e917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:16:26.988174 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.988147    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ab0bb0fe-4b29-4a32-b449-78ed1638b70b" (UID: "ab0bb0fe-4b29-4a32-b449-78ed1638b70b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:26.989745 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:26.989721    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-kube-api-access-q5wrx" (OuterVolumeSpecName: "kube-api-access-q5wrx") pod "ab0bb0fe-4b29-4a32-b449-78ed1638b70b" (UID: "ab0bb0fe-4b29-4a32-b449-78ed1638b70b"). InnerVolumeSpecName "kube-api-access-q5wrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:27.089306 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.089261    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxpvt\" (UniqueName: \"kubernetes.io/projected/373dceac-889f-4689-943c-c3a13893e917-kube-api-access-fxpvt\") pod \"kuadrant-operator-controller-manager-86878c769f-8np8h\" (UID: \"373dceac-889f-4689-943c-c3a13893e917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:16:27.089473 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.089340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/373dceac-889f-4689-943c-c3a13893e917-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8np8h\" (UID: \"373dceac-889f-4689-943c-c3a13893e917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:16:27.089473 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.089383    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wrx\" (UniqueName: \"kubernetes.io/projected/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-kube-api-access-q5wrx\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:16:27.089473 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.089397    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab0bb0fe-4b29-4a32-b449-78ed1638b70b-extensions-socket-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:16:27.089732 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.089711    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/373dceac-889f-4689-943c-c3a13893e917-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8np8h\" (UID: \"373dceac-889f-4689-943c-c3a13893e917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:16:27.096914 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.096885    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxpvt\" (UniqueName: \"kubernetes.io/projected/373dceac-889f-4689-943c-c3a13893e917-kube-api-access-fxpvt\") pod \"kuadrant-operator-controller-manager-86878c769f-8np8h\" (UID: \"373dceac-889f-4689-943c-c3a13893e917\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:16:27.215369 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.215281    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:16:27.344483 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.344453    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"]
Jul 28 20:16:27.348216 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:16:27.348193    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373dceac_889f_4689_943c_c3a13893e917.slice/crio-009bf33ff5c6fdb67407d3b3ecbc8f4a4263ed299bcff90712c545ee52f3befd WatchSource:0}: Error finding container 009bf33ff5c6fdb67407d3b3ecbc8f4a4263ed299bcff90712c545ee52f3befd: Status 404 returned error can't find the container with id 009bf33ff5c6fdb67407d3b3ecbc8f4a4263ed299bcff90712c545ee52f3befd
Jul 28 20:16:27.886970 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.886940    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"
Jul 28 20:16:27.887459 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.886971    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr" event={"ID":"ab0bb0fe-4b29-4a32-b449-78ed1638b70b","Type":"ContainerDied","Data":"f5afeab36d4170c0d03027cf3a8a168f65dee34b6b1b7b37caa00d24f3148602"}
Jul 28 20:16:27.887459 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.887028    2575 scope.go:117] "RemoveContainer" containerID="27735a4cbaf8be20837302fcde88320e93e55b4d993bfe0a752914cdca2a68dc"
Jul 28 20:16:27.888611 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.888582    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h" event={"ID":"373dceac-889f-4689-943c-c3a13893e917","Type":"ContainerStarted","Data":"ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525"}
Jul 28 20:16:27.888701 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.888633    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h" event={"ID":"373dceac-889f-4689-943c-c3a13893e917","Type":"ContainerStarted","Data":"009bf33ff5c6fdb67407d3b3ecbc8f4a4263ed299bcff90712c545ee52f3befd"}
Jul 28 20:16:27.888764 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.888739    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:16:27.906530 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.906494    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h" podStartSLOduration=1.9064806810000001 podStartE2EDuration="1.906480681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:16:27.906248434 +0000 UTC m=+487.520485559" watchObservedRunningTime="2026-07-28 20:16:27.906480681 +0000 UTC m=+487.520717814"
Jul 28 20:16:27.920068 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.920036    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"]
Jul 28 20:16:27.925931 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:27.925909    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9njr"]
Jul 28 20:16:28.838111 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:28.838078    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab0bb0fe-4b29-4a32-b449-78ed1638b70b" path="/var/lib/kubelet/pods/ab0bb0fe-4b29-4a32-b449-78ed1638b70b/volumes"
Jul 28 20:16:38.896213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:38.896181    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:16:43.035846 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:43.035788    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:43.252245 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:43.247846    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:45.570286 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.570249    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"]
Jul 28 20:16:45.570705 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.570467    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:45.570705 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.570622    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab0bb0fe-4b29-4a32-b449-78ed1638b70b" containerName="manager"
Jul 28 20:16:45.570705 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.570635    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab0bb0fe-4b29-4a32-b449-78ed1638b70b" containerName="manager"
Jul 28 20:16:45.570705 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.570685    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab0bb0fe-4b29-4a32-b449-78ed1638b70b" containerName="manager"
Jul 28 20:16:45.573545 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.573524    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:16:45.575617 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.575592    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 20:16:45.575753 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.575659    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t9j99\""
Jul 28 20:16:45.582984 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.582965    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"]
Jul 28 20:16:45.648866 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.648005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzw5n\" (UniqueName: \"kubernetes.io/projected/bc68d866-6fea-4938-b71f-30b9d289d72e-kube-api-access-mzw5n\") pod \"limitador-limitador-64c94fd64f-vrxfd\" (UID: \"bc68d866-6fea-4938-b71f-30b9d289d72e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:16:45.648866 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.648065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc68d866-6fea-4938-b71f-30b9d289d72e-config-file\") pod \"limitador-limitador-64c94fd64f-vrxfd\" (UID: \"bc68d866-6fea-4938-b71f-30b9d289d72e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:16:45.666788 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.666756    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"]
Jul 28 20:16:45.735756 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.735729    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:45.748953 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.748929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzw5n\" (UniqueName: \"kubernetes.io/projected/bc68d866-6fea-4938-b71f-30b9d289d72e-kube-api-access-mzw5n\") pod \"limitador-limitador-64c94fd64f-vrxfd\" (UID: \"bc68d866-6fea-4938-b71f-30b9d289d72e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:16:45.749067 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.748970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc68d866-6fea-4938-b71f-30b9d289d72e-config-file\") pod \"limitador-limitador-64c94fd64f-vrxfd\" (UID: \"bc68d866-6fea-4938-b71f-30b9d289d72e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:16:45.749606 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.749587    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc68d866-6fea-4938-b71f-30b9d289d72e-config-file\") pod \"limitador-limitador-64c94fd64f-vrxfd\" (UID: \"bc68d866-6fea-4938-b71f-30b9d289d72e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:16:45.759410 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.759388    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzw5n\" (UniqueName: \"kubernetes.io/projected/bc68d866-6fea-4938-b71f-30b9d289d72e-kube-api-access-mzw5n\") pod \"limitador-limitador-64c94fd64f-vrxfd\" (UID: \"bc68d866-6fea-4938-b71f-30b9d289d72e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:16:45.836914 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.836828    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:45.884092 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:45.884053    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:16:46.028461 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.028436    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"]
Jul 28 20:16:46.031390 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:16:46.031363    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc68d866_6fea_4938_b71f_30b9d289d72e.slice/crio-c1e63f8280b73dc091f957e8bfd2a10c4346934eb96eeaba7920f9e7e6d8a8ff WatchSource:0}: Error finding container c1e63f8280b73dc091f957e8bfd2a10c4346934eb96eeaba7920f9e7e6d8a8ff: Status 404 returned error can't find the container with id c1e63f8280b73dc091f957e8bfd2a10c4346934eb96eeaba7920f9e7e6d8a8ff
Jul 28 20:16:46.365985 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.365952    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-ptgl2"]
Jul 28 20:16:46.370384 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.370363    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ptgl2"
Jul 28 20:16:46.372375 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.372358    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-llp4v\""
Jul 28 20:16:46.375628 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.375607    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-ptgl2"]
Jul 28 20:16:46.399830 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.399801    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:46.455011 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.454977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2tht\" (UniqueName: \"kubernetes.io/projected/091807f3-1292-47e3-a025-c3f7fe75bc09-kube-api-access-d2tht\") pod \"authorino-58cd955675-ptgl2\" (UID: \"091807f3-1292-47e3-a025-c3f7fe75bc09\") " pod="kuadrant-system/authorino-58cd955675-ptgl2"
Jul 28 20:16:46.466278 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.466220    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:46.556380 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.556344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2tht\" (UniqueName: \"kubernetes.io/projected/091807f3-1292-47e3-a025-c3f7fe75bc09-kube-api-access-d2tht\") pod \"authorino-58cd955675-ptgl2\" (UID: \"091807f3-1292-47e3-a025-c3f7fe75bc09\") " pod="kuadrant-system/authorino-58cd955675-ptgl2"
Jul 28 20:16:46.566657 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.566631    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2tht\" (UniqueName: \"kubernetes.io/projected/091807f3-1292-47e3-a025-c3f7fe75bc09-kube-api-access-d2tht\") pod \"authorino-58cd955675-ptgl2\" (UID: \"091807f3-1292-47e3-a025-c3f7fe75bc09\") " pod="kuadrant-system/authorino-58cd955675-ptgl2"
Jul 28 20:16:46.680446 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.680406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ptgl2"
Jul 28 20:16:46.836156 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:16:46.836124    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod091807f3_1292_47e3_a025_c3f7fe75bc09.slice/crio-138286cb3df5cacd3588e4f7153b3ee020a6e6fb52fdf50ff8f003bb48c3117e WatchSource:0}: Error finding container 138286cb3df5cacd3588e4f7153b3ee020a6e6fb52fdf50ff8f003bb48c3117e: Status 404 returned error can't find the container with id 138286cb3df5cacd3588e4f7153b3ee020a6e6fb52fdf50ff8f003bb48c3117e
Jul 28 20:16:46.839541 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.839513    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-ptgl2"]
Jul 28 20:16:46.975456 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.975372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ptgl2" event={"ID":"091807f3-1292-47e3-a025-c3f7fe75bc09","Type":"ContainerStarted","Data":"138286cb3df5cacd3588e4f7153b3ee020a6e6fb52fdf50ff8f003bb48c3117e"}
Jul 28 20:16:46.977099 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:46.977071    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd" event={"ID":"bc68d866-6fea-4938-b71f-30b9d289d72e","Type":"ContainerStarted","Data":"c1e63f8280b73dc091f957e8bfd2a10c4346934eb96eeaba7920f9e7e6d8a8ff"}
Jul 28 20:16:48.570523 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:48.570492    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:51.000215 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:51.000173    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd" event={"ID":"bc68d866-6fea-4938-b71f-30b9d289d72e","Type":"ContainerStarted","Data":"5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809"}
Jul 28 20:16:51.000642 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:51.000270    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:16:51.001535 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:51.001511    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ptgl2" event={"ID":"091807f3-1292-47e3-a025-c3f7fe75bc09","Type":"ContainerStarted","Data":"9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9"}
Jul 28 20:16:51.016177 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:51.016124    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd" podStartSLOduration=1.476838076 podStartE2EDuration="6.016109385s" totalImagesPullingTime="4.539271309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:16:51.014213139 +0000 UTC m=+510.628450288" watchObservedRunningTime="2026-07-28 20:16:51.016109385 +0000 UTC m=+510.630346518"
Jul 28 20:16:51.028307 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:51.028252    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-ptgl2" podStartSLOduration=1.23439503 podStartE2EDuration="5.028213579s" totalImagesPullingTime="3.793818549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:16:51.027382785 +0000 UTC m=+510.641619913" watchObservedRunningTime="2026-07-28 20:16:51.028213579 +0000 UTC m=+510.642450713"
Jul 28 20:16:51.856737 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:51.856705    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-ptgl2"]
Jul 28 20:16:53.010071 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:53.010033    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-ptgl2" podUID="091807f3-1292-47e3-a025-c3f7fe75bc09" containerName="authorino" containerID="cri-o://9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9" gracePeriod=30
Jul 28 20:16:53.260924 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:53.260869    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ptgl2"
Jul 28 20:16:53.416185 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:53.416150    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2tht\" (UniqueName: \"kubernetes.io/projected/091807f3-1292-47e3-a025-c3f7fe75bc09-kube-api-access-d2tht\") pod \"091807f3-1292-47e3-a025-c3f7fe75bc09\" (UID: \"091807f3-1292-47e3-a025-c3f7fe75bc09\") "
Jul 28 20:16:53.418196 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:53.418169    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091807f3-1292-47e3-a025-c3f7fe75bc09-kube-api-access-d2tht" (OuterVolumeSpecName: "kube-api-access-d2tht") pod "091807f3-1292-47e3-a025-c3f7fe75bc09" (UID: "091807f3-1292-47e3-a025-c3f7fe75bc09"). InnerVolumeSpecName "kube-api-access-d2tht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:53.516934 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:53.516832    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2tht\" (UniqueName: \"kubernetes.io/projected/091807f3-1292-47e3-a025-c3f7fe75bc09-kube-api-access-d2tht\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:16:54.015351 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:54.015310    2575 generic.go:356] "Generic (PLEG): container finished" podID="091807f3-1292-47e3-a025-c3f7fe75bc09" containerID="9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9" exitCode=0
Jul 28 20:16:54.015351 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:54.015353    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ptgl2"
Jul 28 20:16:54.015837 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:54.015396    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ptgl2" event={"ID":"091807f3-1292-47e3-a025-c3f7fe75bc09","Type":"ContainerDied","Data":"9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9"}
Jul 28 20:16:54.015837 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:54.015440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ptgl2" event={"ID":"091807f3-1292-47e3-a025-c3f7fe75bc09","Type":"ContainerDied","Data":"138286cb3df5cacd3588e4f7153b3ee020a6e6fb52fdf50ff8f003bb48c3117e"}
Jul 28 20:16:54.015837 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:54.015459    2575 scope.go:117] "RemoveContainer" containerID="9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9"
Jul 28 20:16:54.023984 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:54.023967    2575 scope.go:117] "RemoveContainer" containerID="9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9"
Jul 28 20:16:54.024257 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:16:54.024215    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9\": container with ID starting with 9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9 not found: ID does not exist" containerID="9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9"
Jul 28 20:16:54.024355 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:54.024264    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9"} err="failed to get container status \"9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9\": rpc error: code = NotFound desc = could not find container \"9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9\": container with ID starting with 9a8fd13819452257fe6a3966bfbabf7b88f27e88146e2717a91a3d980e5217e9 not found: ID does not exist"
Jul 28 20:16:54.035643 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:54.035619    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-ptgl2"]
Jul 28 20:16:54.039698 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:54.039677    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-ptgl2"]
Jul 28 20:16:54.835615 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:16:54.835581    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="091807f3-1292-47e3-a025-c3f7fe75bc09" path="/var/lib/kubelet/pods/091807f3-1292-47e3-a025-c3f7fe75bc09/volumes"
Jul 28 20:17:02.006806 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:02.006776    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:17:02.853715 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:02.853678    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"]
Jul 28 20:17:02.853932 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:02.853909    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd" podUID="bc68d866-6fea-4938-b71f-30b9d289d72e" containerName="limitador" containerID="cri-o://5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809" gracePeriod=30
Jul 28 20:17:03.403653 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:03.403628    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:17:03.493653 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:03.493556    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc68d866-6fea-4938-b71f-30b9d289d72e-config-file\") pod \"bc68d866-6fea-4938-b71f-30b9d289d72e\" (UID: \"bc68d866-6fea-4938-b71f-30b9d289d72e\") "
Jul 28 20:17:03.493653 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:03.493616    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzw5n\" (UniqueName: \"kubernetes.io/projected/bc68d866-6fea-4938-b71f-30b9d289d72e-kube-api-access-mzw5n\") pod \"bc68d866-6fea-4938-b71f-30b9d289d72e\" (UID: \"bc68d866-6fea-4938-b71f-30b9d289d72e\") "
Jul 28 20:17:03.493938 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:03.493914    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc68d866-6fea-4938-b71f-30b9d289d72e-config-file" (OuterVolumeSpecName: "config-file") pod "bc68d866-6fea-4938-b71f-30b9d289d72e" (UID: "bc68d866-6fea-4938-b71f-30b9d289d72e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:17:03.495605 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:03.495582    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc68d866-6fea-4938-b71f-30b9d289d72e-kube-api-access-mzw5n" (OuterVolumeSpecName: "kube-api-access-mzw5n") pod "bc68d866-6fea-4938-b71f-30b9d289d72e" (UID: "bc68d866-6fea-4938-b71f-30b9d289d72e"). InnerVolumeSpecName "kube-api-access-mzw5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:17:03.594118 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:03.594077    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzw5n\" (UniqueName: \"kubernetes.io/projected/bc68d866-6fea-4938-b71f-30b9d289d72e-kube-api-access-mzw5n\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:17:03.594118 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:03.594111    2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc68d866-6fea-4938-b71f-30b9d289d72e-config-file\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:17:04.060213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:04.060178    2575 generic.go:356] "Generic (PLEG): container finished" podID="bc68d866-6fea-4938-b71f-30b9d289d72e" containerID="5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809" exitCode=0
Jul 28 20:17:04.060436 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:04.060251    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"
Jul 28 20:17:04.060436 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:04.060255    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd" event={"ID":"bc68d866-6fea-4938-b71f-30b9d289d72e","Type":"ContainerDied","Data":"5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809"}
Jul 28 20:17:04.060436 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:04.060367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vrxfd" event={"ID":"bc68d866-6fea-4938-b71f-30b9d289d72e","Type":"ContainerDied","Data":"c1e63f8280b73dc091f957e8bfd2a10c4346934eb96eeaba7920f9e7e6d8a8ff"}
Jul 28 20:17:04.060436 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:04.060389    2575 scope.go:117] "RemoveContainer" containerID="5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809"
Jul 28 20:17:04.069405 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:04.069383    2575 scope.go:117] "RemoveContainer" containerID="5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809"
Jul 28 20:17:04.069673 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:17:04.069652    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809\": container with ID starting with 5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809 not found: ID does not exist" containerID="5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809"
Jul 28 20:17:04.069720 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:04.069680    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809"} err="failed to get container status \"5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809\": rpc error: code = NotFound desc = could not find container \"5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809\": container with ID starting with 5ff06720ed84cfa296e0b766020f362babc2f357fe9be740e25dab611b6c2809 not found: ID does not exist"
Jul 28 20:17:04.081606 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:04.081578    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"]
Jul 28 20:17:04.085084 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:04.085057    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vrxfd"]
Jul 28 20:17:04.834965 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:04.834935    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc68d866-6fea-4938-b71f-30b9d289d72e" path="/var/lib/kubelet/pods/bc68d866-6fea-4938-b71f-30b9d289d72e/volumes"
Jul 28 20:17:06.808235 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:06.808203    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:06.917877 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:06.917841    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:07.271888 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.271852    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"]
Jul 28 20:17:07.272166 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.272154    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc68d866-6fea-4938-b71f-30b9d289d72e" containerName="limitador"
Jul 28 20:17:07.272213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.272169    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc68d866-6fea-4938-b71f-30b9d289d72e" containerName="limitador"
Jul 28 20:17:07.272213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.272182    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091807f3-1292-47e3-a025-c3f7fe75bc09" containerName="authorino"
Jul 28 20:17:07.272213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.272188    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="091807f3-1292-47e3-a025-c3f7fe75bc09" containerName="authorino"
Jul 28 20:17:07.272330 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.272257    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc68d866-6fea-4938-b71f-30b9d289d72e" containerName="limitador"
Jul 28 20:17:07.272330 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.272266    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="091807f3-1292-47e3-a025-c3f7fe75bc09" containerName="authorino"
Jul 28 20:17:07.276337 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.276319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"
Jul 28 20:17:07.278675 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.278648    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:17:07.278855 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.278790    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 28 20:17:07.278933 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.278844    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:17:07.279324 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.279307    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ncbr2\""
Jul 28 20:17:07.282270 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.282252    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"]
Jul 28 20:17:07.301379 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.301347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:07.318571 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.318541    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1464ff9a-b5f8-4849-a493-2b58539d7284-data\") pod \"postgres-69fb4594cf-pm6q4\" (UID: \"1464ff9a-b5f8-4849-a493-2b58539d7284\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"
Jul 28 20:17:07.318690 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.318596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98vwp\" (UniqueName: \"kubernetes.io/projected/1464ff9a-b5f8-4849-a493-2b58539d7284-kube-api-access-98vwp\") pod \"postgres-69fb4594cf-pm6q4\" (UID: \"1464ff9a-b5f8-4849-a493-2b58539d7284\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"
Jul 28 20:17:07.419666 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.419623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1464ff9a-b5f8-4849-a493-2b58539d7284-data\") pod \"postgres-69fb4594cf-pm6q4\" (UID: \"1464ff9a-b5f8-4849-a493-2b58539d7284\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"
Jul 28 20:17:07.419870 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.419684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98vwp\" (UniqueName: \"kubernetes.io/projected/1464ff9a-b5f8-4849-a493-2b58539d7284-kube-api-access-98vwp\") pod \"postgres-69fb4594cf-pm6q4\" (UID: \"1464ff9a-b5f8-4849-a493-2b58539d7284\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"
Jul 28 20:17:07.420063 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.420040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1464ff9a-b5f8-4849-a493-2b58539d7284-data\") pod \"postgres-69fb4594cf-pm6q4\" (UID: \"1464ff9a-b5f8-4849-a493-2b58539d7284\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"
Jul 28 20:17:07.427899 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.427865    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98vwp\" (UniqueName: \"kubernetes.io/projected/1464ff9a-b5f8-4849-a493-2b58539d7284-kube-api-access-98vwp\") pod \"postgres-69fb4594cf-pm6q4\" (UID: \"1464ff9a-b5f8-4849-a493-2b58539d7284\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"
Jul 28 20:17:07.587839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.587730    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"
Jul 28 20:17:07.723323 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:07.723288    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"]
Jul 28 20:17:07.726645 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:17:07.726617    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1464ff9a_b5f8_4849_a493_2b58539d7284.slice/crio-a986c28040349c4c0f2e3250acce9ce7e6fa3a9aeef71d64a12140ae268468e4 WatchSource:0}: Error finding container a986c28040349c4c0f2e3250acce9ce7e6fa3a9aeef71d64a12140ae268468e4: Status 404 returned error can't find the container with id a986c28040349c4c0f2e3250acce9ce7e6fa3a9aeef71d64a12140ae268468e4
Jul 28 20:17:08.079555 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:08.079518    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4" event={"ID":"1464ff9a-b5f8-4849-a493-2b58539d7284","Type":"ContainerStarted","Data":"a986c28040349c4c0f2e3250acce9ce7e6fa3a9aeef71d64a12140ae268468e4"}
Jul 28 20:17:19.134648 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:19.134608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4" event={"ID":"1464ff9a-b5f8-4849-a493-2b58539d7284","Type":"ContainerStarted","Data":"b499afd4667bcabf5bc05d1ae53ee71f7b86ad6f431bda08a679aaeaad997c89"}
Jul 28 20:17:19.135051 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:19.134734    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"
Jul 28 20:17:19.150293 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:19.150215    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4" podStartSLOduration=1.729582789 podStartE2EDuration="12.150198343s" totalImagesPullingTime="10.420615554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:19.148836569 +0000 UTC m=+538.763073704" watchObservedRunningTime="2026-07-28 20:17:19.150198343 +0000 UTC m=+538.764435476"
Jul 28 20:17:25.167753 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.167723    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pm6q4"
Jul 28 20:17:25.490757 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.490678    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:25.589035 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.588997    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:25.730425 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.730388    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bnqmn"]
Jul 28 20:17:25.735061 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.735037    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bnqmn"
Jul 28 20:17:25.737173 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.737150    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-llp4v\""
Jul 28 20:17:25.738070 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.738043    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bnqmn"]
Jul 28 20:17:25.752740 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.752713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czxx5\" (UniqueName: \"kubernetes.io/projected/276ef663-bc90-44c2-b0e6-6f85b380a481-kube-api-access-czxx5\") pod \"authorino-5d4b67c446-bnqmn\" (UID: \"276ef663-bc90-44c2-b0e6-6f85b380a481\") " pod="kuadrant-system/authorino-5d4b67c446-bnqmn"
Jul 28 20:17:25.853513 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.853474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czxx5\" (UniqueName: \"kubernetes.io/projected/276ef663-bc90-44c2-b0e6-6f85b380a481-kube-api-access-czxx5\") pod \"authorino-5d4b67c446-bnqmn\" (UID: \"276ef663-bc90-44c2-b0e6-6f85b380a481\") " pod="kuadrant-system/authorino-5d4b67c446-bnqmn"
Jul 28 20:17:25.861058 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.861024    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czxx5\" (UniqueName: \"kubernetes.io/projected/276ef663-bc90-44c2-b0e6-6f85b380a481-kube-api-access-czxx5\") pod \"authorino-5d4b67c446-bnqmn\" (UID: \"276ef663-bc90-44c2-b0e6-6f85b380a481\") " pod="kuadrant-system/authorino-5d4b67c446-bnqmn"
Jul 28 20:17:25.970621 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.970585    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bnqmn"]
Jul 28 20:17:25.970821 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.970806    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bnqmn"
Jul 28 20:17:25.998070 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.993131    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69dd6c8fdf-rv8qt"]
Jul 28 20:17:25.998508 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:25.998484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt"
Jul 28 20:17:26.003799 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:26.003774    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69dd6c8fdf-rv8qt"]
Jul 28 20:17:26.055714 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:26.055383    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxg5l\" (UniqueName: \"kubernetes.io/projected/eddcba26-37f5-4842-b84d-dd7120d2fef1-kube-api-access-pxg5l\") pod \"authorino-69dd6c8fdf-rv8qt\" (UID: \"eddcba26-37f5-4842-b84d-dd7120d2fef1\") " pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt"
Jul 28 20:17:26.099036 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:26.099005    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bnqmn"]
Jul 28 20:17:26.101863 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:17:26.101826    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod276ef663_bc90_44c2_b0e6_6f85b380a481.slice/crio-87b6e7bb1be99e5bdd6d7f331fd7e288934c137e76ceaa89011b27f3d34500dc WatchSource:0}: Error finding container 87b6e7bb1be99e5bdd6d7f331fd7e288934c137e76ceaa89011b27f3d34500dc: Status 404 returned error can't find the container with id 87b6e7bb1be99e5bdd6d7f331fd7e288934c137e76ceaa89011b27f3d34500dc
Jul 28 20:17:26.156851 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:26.156803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxg5l\" (UniqueName: \"kubernetes.io/projected/eddcba26-37f5-4842-b84d-dd7120d2fef1-kube-api-access-pxg5l\") pod \"authorino-69dd6c8fdf-rv8qt\" (UID: \"eddcba26-37f5-4842-b84d-dd7120d2fef1\") " pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt"
Jul 28 20:17:26.165314 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:26.165288    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxg5l\" (UniqueName: \"kubernetes.io/projected/eddcba26-37f5-4842-b84d-dd7120d2fef1-kube-api-access-pxg5l\") pod \"authorino-69dd6c8fdf-rv8qt\" (UID: \"eddcba26-37f5-4842-b84d-dd7120d2fef1\") " pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt"
Jul 28 20:17:26.165997 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:26.165972    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bnqmn" event={"ID":"276ef663-bc90-44c2-b0e6-6f85b380a481","Type":"ContainerStarted","Data":"87b6e7bb1be99e5bdd6d7f331fd7e288934c137e76ceaa89011b27f3d34500dc"}
Jul 28 20:17:26.266214 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:26.266133    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69dd6c8fdf-rv8qt"]
Jul 28 20:17:26.266595 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:26.266384    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt"
Jul 28 20:17:26.421760 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:26.421708    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69dd6c8fdf-rv8qt"]
Jul 28 20:17:26.425483 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:17:26.425460    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeddcba26_37f5_4842_b84d_dd7120d2fef1.slice/crio-1ac673d620db67b8566df0c5a77bf45c5e567f980b52498ce3346be609becefc WatchSource:0}: Error finding container 1ac673d620db67b8566df0c5a77bf45c5e567f980b52498ce3346be609becefc: Status 404 returned error can't find the container with id 1ac673d620db67b8566df0c5a77bf45c5e567f980b52498ce3346be609becefc
Jul 28 20:17:27.171799 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.171757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bnqmn" event={"ID":"276ef663-bc90-44c2-b0e6-6f85b380a481","Type":"ContainerStarted","Data":"4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631"}
Jul 28 20:17:27.171799 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.171760    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-bnqmn" podUID="276ef663-bc90-44c2-b0e6-6f85b380a481" containerName="authorino" containerID="cri-o://4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631" gracePeriod=30
Jul 28 20:17:27.173425 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.173401    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt" event={"ID":"eddcba26-37f5-4842-b84d-dd7120d2fef1","Type":"ContainerStarted","Data":"8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2"}
Jul 28 20:17:27.173531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.173433    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt" event={"ID":"eddcba26-37f5-4842-b84d-dd7120d2fef1","Type":"ContainerStarted","Data":"1ac673d620db67b8566df0c5a77bf45c5e567f980b52498ce3346be609becefc"}
Jul 28 20:17:27.173531 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.173457    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt" podUID="eddcba26-37f5-4842-b84d-dd7120d2fef1" containerName="authorino" containerID="cri-o://8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2" gracePeriod=30
Jul 28 20:17:27.186188 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.186148    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-bnqmn" podStartSLOduration=1.876714027 podStartE2EDuration="2.186133645s" totalImagesPullingTime="309.419618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:27.185393288 +0000 UTC m=+546.799630420" watchObservedRunningTime="2026-07-28 20:17:27.186133645 +0000 UTC m=+546.800370778"
Jul 28 20:17:27.200350 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.200312    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt" podStartSLOduration=1.80371859 podStartE2EDuration="2.200298437s" totalImagesPullingTime="396.579847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:27.198756471 +0000 UTC m=+546.812993604" watchObservedRunningTime="2026-07-28 20:17:27.200298437 +0000 UTC m=+546.814535570"
Jul 28 20:17:27.432588 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.432565    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt"
Jul 28 20:17:27.435489 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.435469    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bnqmn"
Jul 28 20:17:27.467639 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.467607    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxg5l\" (UniqueName: \"kubernetes.io/projected/eddcba26-37f5-4842-b84d-dd7120d2fef1-kube-api-access-pxg5l\") pod \"eddcba26-37f5-4842-b84d-dd7120d2fef1\" (UID: \"eddcba26-37f5-4842-b84d-dd7120d2fef1\") "
Jul 28 20:17:27.467791 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.467651    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czxx5\" (UniqueName: \"kubernetes.io/projected/276ef663-bc90-44c2-b0e6-6f85b380a481-kube-api-access-czxx5\") pod \"276ef663-bc90-44c2-b0e6-6f85b380a481\" (UID: \"276ef663-bc90-44c2-b0e6-6f85b380a481\") "
Jul 28 20:17:27.469632 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.469604    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddcba26-37f5-4842-b84d-dd7120d2fef1-kube-api-access-pxg5l" (OuterVolumeSpecName: "kube-api-access-pxg5l") pod "eddcba26-37f5-4842-b84d-dd7120d2fef1" (UID: "eddcba26-37f5-4842-b84d-dd7120d2fef1"). InnerVolumeSpecName "kube-api-access-pxg5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:17:27.469632 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.469619    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276ef663-bc90-44c2-b0e6-6f85b380a481-kube-api-access-czxx5" (OuterVolumeSpecName: "kube-api-access-czxx5") pod "276ef663-bc90-44c2-b0e6-6f85b380a481" (UID: "276ef663-bc90-44c2-b0e6-6f85b380a481"). InnerVolumeSpecName "kube-api-access-czxx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:17:27.568878 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.568840    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxg5l\" (UniqueName: \"kubernetes.io/projected/eddcba26-37f5-4842-b84d-dd7120d2fef1-kube-api-access-pxg5l\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:17:27.568878 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:27.568871    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czxx5\" (UniqueName: \"kubernetes.io/projected/276ef663-bc90-44c2-b0e6-6f85b380a481-kube-api-access-czxx5\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:17:28.178716 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.178678    2575 generic.go:356] "Generic (PLEG): container finished" podID="eddcba26-37f5-4842-b84d-dd7120d2fef1" containerID="8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2" exitCode=0
Jul 28 20:17:28.178902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.178723    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt"
Jul 28 20:17:28.178902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.178761    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt" event={"ID":"eddcba26-37f5-4842-b84d-dd7120d2fef1","Type":"ContainerDied","Data":"8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2"}
Jul 28 20:17:28.178902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.178804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69dd6c8fdf-rv8qt" event={"ID":"eddcba26-37f5-4842-b84d-dd7120d2fef1","Type":"ContainerDied","Data":"1ac673d620db67b8566df0c5a77bf45c5e567f980b52498ce3346be609becefc"}
Jul 28 20:17:28.178902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.178822    2575 scope.go:117] "RemoveContainer" containerID="8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2"
Jul 28 20:17:28.180138 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.180024    2575 generic.go:356] "Generic (PLEG): container finished" podID="276ef663-bc90-44c2-b0e6-6f85b380a481" containerID="4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631" exitCode=0
Jul 28 20:17:28.180138 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.180062    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bnqmn" event={"ID":"276ef663-bc90-44c2-b0e6-6f85b380a481","Type":"ContainerDied","Data":"4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631"}
Jul 28 20:17:28.180138 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.180077    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bnqmn"
Jul 28 20:17:28.180138 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.180090    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bnqmn" event={"ID":"276ef663-bc90-44c2-b0e6-6f85b380a481","Type":"ContainerDied","Data":"87b6e7bb1be99e5bdd6d7f331fd7e288934c137e76ceaa89011b27f3d34500dc"}
Jul 28 20:17:28.187908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.187889    2575 scope.go:117] "RemoveContainer" containerID="8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2"
Jul 28 20:17:28.188219 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:17:28.188197    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2\": container with ID starting with 8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2 not found: ID does not exist" containerID="8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2"
Jul 28 20:17:28.188336 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.188238    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2"} err="failed to get container status \"8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2\": rpc error: code = NotFound desc = could not find container \"8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2\": container with ID starting with 8410e86fea775f47dd99d57461fe2f71cb0ff49b52e007d1db799833b6d368b2 not found: ID does not exist"
Jul 28 20:17:28.188336 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.188256    2575 scope.go:117] "RemoveContainer" containerID="4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631"
Jul 28 20:17:28.195700 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.195683    2575 scope.go:117] "RemoveContainer" containerID="4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631"
Jul 28 20:17:28.195946 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:17:28.195925    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631\": container with ID starting with 4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631 not found: ID does not exist" containerID="4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631"
Jul 28 20:17:28.195986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.195952    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631"} err="failed to get container status \"4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631\": rpc error: code = NotFound desc = could not find container \"4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631\": container with ID starting with 4c9c77361597d5dd1f5312bf99b6fd91ba0bc622bfce77fb92d6b9dbea391631 not found: ID does not exist"
Jul 28 20:17:28.198986 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.198967    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69dd6c8fdf-rv8qt"]
Jul 28 20:17:28.207889 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.207867    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69dd6c8fdf-rv8qt"]
Jul 28 20:17:28.219750 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.219726    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bnqmn"]
Jul 28 20:17:28.225628 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.225607    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bnqmn"]
Jul 28 20:17:28.835439 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.835407    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276ef663-bc90-44c2-b0e6-6f85b380a481" path="/var/lib/kubelet/pods/276ef663-bc90-44c2-b0e6-6f85b380a481/volumes"
Jul 28 20:17:28.835923 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:28.835856    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eddcba26-37f5-4842-b84d-dd7120d2fef1" path="/var/lib/kubelet/pods/eddcba26-37f5-4842-b84d-dd7120d2fef1/volumes"
Jul 28 20:17:32.802893 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.802858    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:32.811317 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.811290    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67fb85b555-2xbpq"]
Jul 28 20:17:32.811781 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.811767    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276ef663-bc90-44c2-b0e6-6f85b380a481" containerName="authorino"
Jul 28 20:17:32.811829 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.811785    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="276ef663-bc90-44c2-b0e6-6f85b380a481" containerName="authorino"
Jul 28 20:17:32.811829 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.811808    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eddcba26-37f5-4842-b84d-dd7120d2fef1" containerName="authorino"
Jul 28 20:17:32.811829 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.811816    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eddcba26-37f5-4842-b84d-dd7120d2fef1" containerName="authorino"
Jul 28 20:17:32.811919 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.811898    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eddcba26-37f5-4842-b84d-dd7120d2fef1" containerName="authorino"
Jul 28 20:17:32.811919 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.811913    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="276ef663-bc90-44c2-b0e6-6f85b380a481" containerName="authorino"
Jul 28 20:17:32.815234 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.815207    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:32.822563 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.822540    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 20:17:32.822664 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.822542    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 20:17:32.822664 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.822563    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fbtrn\""
Jul 28 20:17:32.837793 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.837766    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67fb85b555-2xbpq"]
Jul 28 20:17:32.897618 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.897582    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:32.913725 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.913700    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctsg4\" (UniqueName: \"kubernetes.io/projected/c01f9f24-d484-4eea-b741-045c18023a6a-kube-api-access-ctsg4\") pod \"maas-controller-67fb85b555-2xbpq\" (UID: \"c01f9f24-d484-4eea-b741-045c18023a6a\") " pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:32.913824 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.913735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c01f9f24-d484-4eea-b741-045c18023a6a-webhook-cert\") pod \"maas-controller-67fb85b555-2xbpq\" (UID: \"c01f9f24-d484-4eea-b741-045c18023a6a\") " pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:32.985742 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.985711    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:32.990172 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.990144    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-kq6lp"]
Jul 28 20:17:32.992655 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:32.992634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:33.004779 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.004755    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-kq6lp"]
Jul 28 20:17:33.015242 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.015195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctsg4\" (UniqueName: \"kubernetes.io/projected/c01f9f24-d484-4eea-b741-045c18023a6a-kube-api-access-ctsg4\") pod \"maas-controller-67fb85b555-2xbpq\" (UID: \"c01f9f24-d484-4eea-b741-045c18023a6a\") " pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:33.015383 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.015334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c01f9f24-d484-4eea-b741-045c18023a6a-webhook-cert\") pod \"maas-controller-67fb85b555-2xbpq\" (UID: \"c01f9f24-d484-4eea-b741-045c18023a6a\") " pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:33.015492 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:17:33.015464    2575 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 28 20:17:33.015616 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:17:33.015532    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01f9f24-d484-4eea-b741-045c18023a6a-webhook-cert podName:c01f9f24-d484-4eea-b741-045c18023a6a nodeName:}" failed. No retries permitted until 2026-07-28 20:17:33.515511589 +0000 UTC m=+553.129748706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/c01f9f24-d484-4eea-b741-045c18023a6a-webhook-cert") pod "maas-controller-67fb85b555-2xbpq" (UID: "c01f9f24-d484-4eea-b741-045c18023a6a") : secret "maas-controller-webhook-cert" not found
Jul 28 20:17:33.015730 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.015709    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-webhook-cert\") pod \"maas-controller-5dc5fc9c88-kq6lp\" (UID: \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\") " pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:33.015992 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.015893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvd9\" (UniqueName: \"kubernetes.io/projected/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-kube-api-access-5mvd9\") pod \"maas-controller-5dc5fc9c88-kq6lp\" (UID: \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\") " pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:33.025519 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.025492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctsg4\" (UniqueName: \"kubernetes.io/projected/c01f9f24-d484-4eea-b741-045c18023a6a-kube-api-access-ctsg4\") pod \"maas-controller-67fb85b555-2xbpq\" (UID: \"c01f9f24-d484-4eea-b741-045c18023a6a\") " pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:33.116580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.116485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-webhook-cert\") pod \"maas-controller-5dc5fc9c88-kq6lp\" (UID: \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\") " pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:33.116580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.116546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvd9\" (UniqueName: \"kubernetes.io/projected/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-kube-api-access-5mvd9\") pod \"maas-controller-5dc5fc9c88-kq6lp\" (UID: \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\") " pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:33.116766 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:17:33.116650    2575 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 28 20:17:33.116766 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:17:33.116730    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-webhook-cert podName:3e9cbe82-6bc3-4987-9bc2-bbf7111cff37 nodeName:}" failed. No retries permitted until 2026-07-28 20:17:33.616708939 +0000 UTC m=+553.230946054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-webhook-cert") pod "maas-controller-5dc5fc9c88-kq6lp" (UID: "3e9cbe82-6bc3-4987-9bc2-bbf7111cff37") : secret "maas-controller-webhook-cert" not found
Jul 28 20:17:33.125829 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.125798    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvd9\" (UniqueName: \"kubernetes.io/projected/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-kube-api-access-5mvd9\") pod \"maas-controller-5dc5fc9c88-kq6lp\" (UID: \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\") " pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:33.364595 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.364564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:33.519466 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.519429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c01f9f24-d484-4eea-b741-045c18023a6a-webhook-cert\") pod \"maas-controller-67fb85b555-2xbpq\" (UID: \"c01f9f24-d484-4eea-b741-045c18023a6a\") " pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:33.521815 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.521784    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c01f9f24-d484-4eea-b741-045c18023a6a-webhook-cert\") pod \"maas-controller-67fb85b555-2xbpq\" (UID: \"c01f9f24-d484-4eea-b741-045c18023a6a\") " pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:33.620812 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.620764    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-webhook-cert\") pod \"maas-controller-5dc5fc9c88-kq6lp\" (UID: \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\") " pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:33.623076 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.623053    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-webhook-cert\") pod \"maas-controller-5dc5fc9c88-kq6lp\" (UID: \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\") " pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:33.725605 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.725566    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:33.855614 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.855560    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67fb85b555-2xbpq"]
Jul 28 20:17:33.858115 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:17:33.858087    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01f9f24_d484_4eea_b741_045c18023a6a.slice/crio-6208fa2e4e9fe314afb30bc313c5f009d6a863ae828f61f494d507a5dd80bfb5 WatchSource:0}: Error finding container 6208fa2e4e9fe314afb30bc313c5f009d6a863ae828f61f494d507a5dd80bfb5: Status 404 returned error can't find the container with id 6208fa2e4e9fe314afb30bc313c5f009d6a863ae828f61f494d507a5dd80bfb5
Jul 28 20:17:33.902564 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:33.902529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:34.030280 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:34.030211    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-kq6lp"]
Jul 28 20:17:34.032752 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:17:34.032720    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9cbe82_6bc3_4987_9bc2_bbf7111cff37.slice/crio-e28f1e26623fe25a2a4f930df9edb7c9012f4849a17dc96c2e5f7855a8a9ea1b WatchSource:0}: Error finding container e28f1e26623fe25a2a4f930df9edb7c9012f4849a17dc96c2e5f7855a8a9ea1b: Status 404 returned error can't find the container with id e28f1e26623fe25a2a4f930df9edb7c9012f4849a17dc96c2e5f7855a8a9ea1b
Jul 28 20:17:34.209968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:34.209936    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp" event={"ID":"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37","Type":"ContainerStarted","Data":"e28f1e26623fe25a2a4f930df9edb7c9012f4849a17dc96c2e5f7855a8a9ea1b"}
Jul 28 20:17:34.211051 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:34.211023    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67fb85b555-2xbpq" event={"ID":"c01f9f24-d484-4eea-b741-045c18023a6a","Type":"ContainerStarted","Data":"6208fa2e4e9fe314afb30bc313c5f009d6a863ae828f61f494d507a5dd80bfb5"}
Jul 28 20:17:36.557167 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:36.557124    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:36.650795 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:36.650742    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:37.228724 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:37.228687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67fb85b555-2xbpq" event={"ID":"c01f9f24-d484-4eea-b741-045c18023a6a","Type":"ContainerStarted","Data":"3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505"}
Jul 28 20:17:37.228946 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:37.228831    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:37.230285 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:37.230260    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp" event={"ID":"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37","Type":"ContainerStarted","Data":"bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8"}
Jul 28 20:17:37.230463 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:37.230446    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:37.244991 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:37.244942    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67fb85b555-2xbpq" podStartSLOduration=2.725567009 podStartE2EDuration="5.24492896s" totalImagesPullingTime="2.519361951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:37.244132447 +0000 UTC m=+556.858369582" watchObservedRunningTime="2026-07-28 20:17:37.24492896 +0000 UTC m=+556.859166132"
Jul 28 20:17:37.259838 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:37.259778    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp" podStartSLOduration=2.908298117 podStartE2EDuration="5.259760525s" totalImagesPullingTime="2.351462408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:37.259036532 +0000 UTC m=+556.873273667" watchObservedRunningTime="2026-07-28 20:17:37.259760525 +0000 UTC m=+556.873997659"
Jul 28 20:17:48.237944 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.237911    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:17:48.238357 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.237988    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:48.300248 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.300198    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67fb85b555-2xbpq"]
Jul 28 20:17:48.300496 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.300465    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-67fb85b555-2xbpq" podUID="c01f9f24-d484-4eea-b741-045c18023a6a" containerName="manager" containerID="cri-o://3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505" gracePeriod=30
Jul 28 20:17:48.554091 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.554062    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:48.651446 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.651387    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctsg4\" (UniqueName: \"kubernetes.io/projected/c01f9f24-d484-4eea-b741-045c18023a6a-kube-api-access-ctsg4\") pod \"c01f9f24-d484-4eea-b741-045c18023a6a\" (UID: \"c01f9f24-d484-4eea-b741-045c18023a6a\") "
Jul 28 20:17:48.651619 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.651460    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c01f9f24-d484-4eea-b741-045c18023a6a-webhook-cert\") pod \"c01f9f24-d484-4eea-b741-045c18023a6a\" (UID: \"c01f9f24-d484-4eea-b741-045c18023a6a\") "
Jul 28 20:17:48.653717 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.653682    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01f9f24-d484-4eea-b741-045c18023a6a-kube-api-access-ctsg4" (OuterVolumeSpecName: "kube-api-access-ctsg4") pod "c01f9f24-d484-4eea-b741-045c18023a6a" (UID: "c01f9f24-d484-4eea-b741-045c18023a6a"). InnerVolumeSpecName "kube-api-access-ctsg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:17:48.653717 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.653702    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01f9f24-d484-4eea-b741-045c18023a6a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c01f9f24-d484-4eea-b741-045c18023a6a" (UID: "c01f9f24-d484-4eea-b741-045c18023a6a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:17:48.751948 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.751908    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctsg4\" (UniqueName: \"kubernetes.io/projected/c01f9f24-d484-4eea-b741-045c18023a6a-kube-api-access-ctsg4\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:17:48.751948 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:48.751941    2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c01f9f24-d484-4eea-b741-045c18023a6a-webhook-cert\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:17:49.285676 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:49.285640    2575 generic.go:356] "Generic (PLEG): container finished" podID="c01f9f24-d484-4eea-b741-045c18023a6a" containerID="3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505" exitCode=0
Jul 28 20:17:49.286119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:49.285690    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67fb85b555-2xbpq"
Jul 28 20:17:49.286119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:49.285730    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67fb85b555-2xbpq" event={"ID":"c01f9f24-d484-4eea-b741-045c18023a6a","Type":"ContainerDied","Data":"3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505"}
Jul 28 20:17:49.286119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:49.285784    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67fb85b555-2xbpq" event={"ID":"c01f9f24-d484-4eea-b741-045c18023a6a","Type":"ContainerDied","Data":"6208fa2e4e9fe314afb30bc313c5f009d6a863ae828f61f494d507a5dd80bfb5"}
Jul 28 20:17:49.286119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:49.285809    2575 scope.go:117] "RemoveContainer" containerID="3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505"
Jul 28 20:17:49.294340 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:49.294313    2575 scope.go:117] "RemoveContainer" containerID="3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505"
Jul 28 20:17:49.294609 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:17:49.294590    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505\": container with ID starting with 3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505 not found: ID does not exist" containerID="3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505"
Jul 28 20:17:49.294659 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:49.294617    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505"} err="failed to get container status \"3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505\": rpc error: code = NotFound desc = could not find container \"3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505\": container with ID starting with 3f60f483846584599d1e64ad43cc47474fda9e99e5585acb12fa041d7a233505 not found: ID does not exist"
Jul 28 20:17:49.302377 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:49.302356    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67fb85b555-2xbpq"]
Jul 28 20:17:49.305599 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:49.305576    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67fb85b555-2xbpq"]
Jul 28 20:17:50.835746 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:17:50.835713    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c01f9f24-d484-4eea-b741-045c18023a6a" path="/var/lib/kubelet/pods/c01f9f24-d484-4eea-b741-045c18023a6a/volumes"
Jul 28 20:18:10.338830 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:10.338789    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:10.438340 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:10.438301    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:10.943921 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:10.943886    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:11.083019 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.082986    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"]
Jul 28 20:18:11.083394 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.083379    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01f9f24-d484-4eea-b741-045c18023a6a" containerName="manager"
Jul 28 20:18:11.083450 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.083395    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01f9f24-d484-4eea-b741-045c18023a6a" containerName="manager"
Jul 28 20:18:11.083450 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.083446    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01f9f24-d484-4eea-b741-045c18023a6a" containerName="manager"
Jul 28 20:18:11.110646 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.110618    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"]
Jul 28 20:18:11.110824 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.110722    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.112916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.112892    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9mbpk\""
Jul 28 20:18:11.113035 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.112973    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 28 20:18:11.167431 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.167398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:11.243696 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.243607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h6cf\" (UniqueName: \"kubernetes.io/projected/45ba6fba-6845-4fd2-8248-b0b5bfd51fe2-kube-api-access-5h6cf\") pod \"maas-api-8b5db48f9-l56bc\" (UID: \"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2\") " pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.243696 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.243646    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ba6fba-6845-4fd2-8248-b0b5bfd51fe2-openshift-service-ca\") pod \"maas-api-8b5db48f9-l56bc\" (UID: \"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2\") " pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.243910 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.243745    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ba6fba-6845-4fd2-8248-b0b5bfd51fe2-maas-api-tls\") pod \"maas-api-8b5db48f9-l56bc\" (UID: \"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2\") " pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.344657 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.344628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h6cf\" (UniqueName: \"kubernetes.io/projected/45ba6fba-6845-4fd2-8248-b0b5bfd51fe2-kube-api-access-5h6cf\") pod \"maas-api-8b5db48f9-l56bc\" (UID: \"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2\") " pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.344657 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.344663    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ba6fba-6845-4fd2-8248-b0b5bfd51fe2-openshift-service-ca\") pod \"maas-api-8b5db48f9-l56bc\" (UID: \"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2\") " pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.345168 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.344690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ba6fba-6845-4fd2-8248-b0b5bfd51fe2-maas-api-tls\") pod \"maas-api-8b5db48f9-l56bc\" (UID: \"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2\") " pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.345360 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.345340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ba6fba-6845-4fd2-8248-b0b5bfd51fe2-openshift-service-ca\") pod \"maas-api-8b5db48f9-l56bc\" (UID: \"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2\") " pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.347074 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.347051    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ba6fba-6845-4fd2-8248-b0b5bfd51fe2-maas-api-tls\") pod \"maas-api-8b5db48f9-l56bc\" (UID: \"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2\") " pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.351893 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.351874    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h6cf\" (UniqueName: \"kubernetes.io/projected/45ba6fba-6845-4fd2-8248-b0b5bfd51fe2-kube-api-access-5h6cf\") pod \"maas-api-8b5db48f9-l56bc\" (UID: \"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2\") " pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.421416 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.421385    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:11.438078 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.438046    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:11.541214 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.541168    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:11.579598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.579573    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"]
Jul 28 20:18:11.581866 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:18:11.581837    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ba6fba_6845_4fd2_8248_b0b5bfd51fe2.slice/crio-1811c34bf3a79bed76b63194659ac53159d8f01b3d10f30819ac1dd08240d1c9 WatchSource:0}: Error finding container 1811c34bf3a79bed76b63194659ac53159d8f01b3d10f30819ac1dd08240d1c9: Status 404 returned error can't find the container with id 1811c34bf3a79bed76b63194659ac53159d8f01b3d10f30819ac1dd08240d1c9
Jul 28 20:18:11.621443 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.621404    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"]
Jul 28 20:18:11.625976 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.625953    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"
Jul 28 20:18:11.628300 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.628247    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 28 20:18:11.628853 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.628714    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-52kzv\""
Jul 28 20:18:11.633549 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.633521    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"]
Jul 28 20:18:11.748218 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.748180    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f553c8-585e-4ab2-91b0-926fc1cf097c-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-9dgtc\" (UID: \"a8f553c8-585e-4ab2-91b0-926fc1cf097c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"
Jul 28 20:18:11.748412 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.748257    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpnmf\" (UniqueName: \"kubernetes.io/projected/a8f553c8-585e-4ab2-91b0-926fc1cf097c-kube-api-access-gpnmf\") pod \"payload-pre-processing-687dbb6cf6-9dgtc\" (UID: \"a8f553c8-585e-4ab2-91b0-926fc1cf097c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"
Jul 28 20:18:11.849371 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.849281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f553c8-585e-4ab2-91b0-926fc1cf097c-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-9dgtc\" (UID: \"a8f553c8-585e-4ab2-91b0-926fc1cf097c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"
Jul 28 20:18:11.849371 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.849334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpnmf\" (UniqueName: \"kubernetes.io/projected/a8f553c8-585e-4ab2-91b0-926fc1cf097c-kube-api-access-gpnmf\") pod \"payload-pre-processing-687dbb6cf6-9dgtc\" (UID: \"a8f553c8-585e-4ab2-91b0-926fc1cf097c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"
Jul 28 20:18:11.850030 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.850011    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f553c8-585e-4ab2-91b0-926fc1cf097c-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-9dgtc\" (UID: \"a8f553c8-585e-4ab2-91b0-926fc1cf097c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"
Jul 28 20:18:11.857886 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.857858    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpnmf\" (UniqueName: \"kubernetes.io/projected/a8f553c8-585e-4ab2-91b0-926fc1cf097c-kube-api-access-gpnmf\") pod \"payload-pre-processing-687dbb6cf6-9dgtc\" (UID: \"a8f553c8-585e-4ab2-91b0-926fc1cf097c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"
Jul 28 20:18:11.938676 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:11.938644    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"
Jul 28 20:18:12.068012 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:12.067841    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"]
Jul 28 20:18:12.070664 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:18:12.070640    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f553c8_585e_4ab2_91b0_926fc1cf097c.slice/crio-bda4825056cfeb2c3f93a8c036b897438c8e8fafacc491d1e735f4857816ab3e WatchSource:0}: Error finding container bda4825056cfeb2c3f93a8c036b897438c8e8fafacc491d1e735f4857816ab3e: Status 404 returned error can't find the container with id bda4825056cfeb2c3f93a8c036b897438c8e8fafacc491d1e735f4857816ab3e
Jul 28 20:18:12.394766 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:12.394727    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc" event={"ID":"a8f553c8-585e-4ab2-91b0-926fc1cf097c","Type":"ContainerStarted","Data":"bda4825056cfeb2c3f93a8c036b897438c8e8fafacc491d1e735f4857816ab3e"}
Jul 28 20:18:12.396023 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:12.396000    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc" event={"ID":"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2","Type":"ContainerStarted","Data":"1811c34bf3a79bed76b63194659ac53159d8f01b3d10f30819ac1dd08240d1c9"}
Jul 28 20:18:14.408218 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:14.408178    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc" event={"ID":"a8f553c8-585e-4ab2-91b0-926fc1cf097c","Type":"ContainerStarted","Data":"cf73391245b389b27e5c869510ba61183b11336ed6e4364284e79c53d993a416"}
Jul 28 20:18:14.408706 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:14.408299    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"
Jul 28 20:18:14.409722 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:14.409702    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc" event={"ID":"45ba6fba-6845-4fd2-8248-b0b5bfd51fe2","Type":"ContainerStarted","Data":"6ed69be52b02215a69b3f8306b2f18d02e1846966f28602982fd95c56efd0b32"}
Jul 28 20:18:14.409835 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:14.409823    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:14.422867 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:14.422823    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc" podStartSLOduration=1.744363135 podStartE2EDuration="3.422811071s" totalImagesPullingTime="1.678447936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:18:14.422511063 +0000 UTC m=+594.036748199" watchObservedRunningTime="2026-07-28 20:18:14.422811071 +0000 UTC m=+594.037048204"
Jul 28 20:18:14.439250 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:14.439199    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc" podStartSLOduration=1.278044951 podStartE2EDuration="3.439185621s" totalImagesPullingTime="2.16114067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:18:14.43769089 +0000 UTC m=+594.051928022" watchObservedRunningTime="2026-07-28 20:18:14.439185621 +0000 UTC m=+594.053422753"
Jul 28 20:18:15.894656 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:15.894621    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:20.420407 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:20.420374    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8b5db48f9-l56bc"
Jul 28 20:18:21.456580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:21.456535    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:21.555094 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:21.555061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:25.416358 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:25.416310    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc" podUID="a8f553c8-585e-4ab2-91b0-926fc1cf097c" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.41:9004: connect: connection refused"
Jul 28 20:18:29.148465 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:29.148428    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:35.414945 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:35.414910    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9dgtc"
Jul 28 20:18:44.597254 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.597151    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"]
Jul 28 20:18:44.600443 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.600425    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.604374 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.604321    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xw4fk\""
Jul 28 20:18:44.604374 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.604321    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 28 20:18:44.604573 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.604494    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 20:18:44.604672 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.604587    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 20:18:44.613466 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.613444    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"]
Jul 28 20:18:44.725110 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.725075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.725110 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.725111    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.725380 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.725133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvh8\" (UniqueName: \"kubernetes.io/projected/63d1ad1d-a6d9-45bc-b189-40bd972dd297-kube-api-access-blvh8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.725380 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.725299    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.725380 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.725328    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d1ad1d-a6d9-45bc-b189-40bd972dd297-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.725380 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.725347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.807186 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.807149    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:44.826313 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.826271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.826313 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.826315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.826564 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.826482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvh8\" (UniqueName: \"kubernetes.io/projected/63d1ad1d-a6d9-45bc-b189-40bd972dd297-kube-api-access-blvh8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.826624 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.826574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.826624 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.826592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d1ad1d-a6d9-45bc-b189-40bd972dd297-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.826624 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.826608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.826784 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.826689    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.826941 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.826925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.826989 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.826925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.828795 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.828756    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d1ad1d-a6d9-45bc-b189-40bd972dd297-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.829146 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.829123    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d1ad1d-a6d9-45bc-b189-40bd972dd297-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.835649 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.835625    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvh8\" (UniqueName: \"kubernetes.io/projected/63d1ad1d-a6d9-45bc-b189-40bd972dd297-kube-api-access-blvh8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2bt48\" (UID: \"63d1ad1d-a6d9-45bc-b189-40bd972dd297\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:44.910859 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:44.910818    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:45.051930 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:45.051902    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"]
Jul 28 20:18:45.054166 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:18:45.054139    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d1ad1d_a6d9_45bc_b189_40bd972dd297.slice/crio-e1f486df40afafaaf51d705accd9e0aa45cfb5b42a28acde34811528c52b5f2d WatchSource:0}: Error finding container e1f486df40afafaaf51d705accd9e0aa45cfb5b42a28acde34811528c52b5f2d: Status 404 returned error can't find the container with id e1f486df40afafaaf51d705accd9e0aa45cfb5b42a28acde34811528c52b5f2d
Jul 28 20:18:45.055724 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:45.055706    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:18:45.549499 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:45.549461    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48" event={"ID":"63d1ad1d-a6d9-45bc-b189-40bd972dd297","Type":"ContainerStarted","Data":"e1f486df40afafaaf51d705accd9e0aa45cfb5b42a28acde34811528c52b5f2d"}
Jul 28 20:18:46.555985 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:46.555942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48" event={"ID":"63d1ad1d-a6d9-45bc-b189-40bd972dd297","Type":"ContainerStarted","Data":"0d48790b62d8f29076d3e8fb73ce192c5f37a123d56f3a9ed7072169394eb993"}
Jul 28 20:18:46.575766 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:46.575712    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48" podStartSLOduration=1.568670027 podStartE2EDuration="2.57569357s" totalImagesPullingTime="1.007023543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:18:46.573843711 +0000 UTC m=+626.188080857" watchObservedRunningTime="2026-07-28 20:18:46.57569357 +0000 UTC m=+626.189930707"
Jul 28 20:18:52.035748 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:52.035713    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:54.911093 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:54.911055    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:54.911711 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:54.911127    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:54.923902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:54.923878    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:18:55.607366 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:18:55.607340    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2bt48"
Jul 28 20:19:00.489426 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.489385    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"]
Jul 28 20:19:00.493123 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.493102    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.495715 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.495696    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 28 20:19:00.505106 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.505084    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"]
Jul 28 20:19:00.572717 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.572682    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.572888 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.572772    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.572888 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.572800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.572888 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.572823    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.573004 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.572892    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822fe93d-94ec-4d7b-96c5-de0735b61c81-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.573004 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.572936    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vdr6\" (UniqueName: \"kubernetes.io/projected/822fe93d-94ec-4d7b-96c5-de0735b61c81-kube-api-access-7vdr6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.601327 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.601298    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:00.673391 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.673348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.673585 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.673402    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.673585 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.673422    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.673585 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.673447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.673585 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.673480    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822fe93d-94ec-4d7b-96c5-de0735b61c81-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.673585 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.673514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vdr6\" (UniqueName: \"kubernetes.io/projected/822fe93d-94ec-4d7b-96c5-de0735b61c81-kube-api-access-7vdr6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.673898 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.673871    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.673968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.673883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.673968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.673951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.675657 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.675638    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/822fe93d-94ec-4d7b-96c5-de0735b61c81-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.675908 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.675886    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822fe93d-94ec-4d7b-96c5-de0735b61c81-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.682359 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.682335    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vdr6\" (UniqueName: \"kubernetes.io/projected/822fe93d-94ec-4d7b-96c5-de0735b61c81-kube-api-access-7vdr6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm\" (UID: \"822fe93d-94ec-4d7b-96c5-de0735b61c81\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.803447 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.803361    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:00.939346 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:00.939264    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"]
Jul 28 20:19:00.942125 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:19:00.942093    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822fe93d_94ec_4d7b_96c5_de0735b61c81.slice/crio-94bab47f8366a81ef99c498941841928551872b94fa61e2d0b10495fffb26762 WatchSource:0}: Error finding container 94bab47f8366a81ef99c498941841928551872b94fa61e2d0b10495fffb26762: Status 404 returned error can't find the container with id 94bab47f8366a81ef99c498941841928551872b94fa61e2d0b10495fffb26762
Jul 28 20:19:01.624104 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:01.624017    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm" event={"ID":"822fe93d-94ec-4d7b-96c5-de0735b61c81","Type":"ContainerStarted","Data":"9cfba9477fad68d44bbabac0a032b3b4108b68661e3d3fa9eb3232e2885977ce"}
Jul 28 20:19:01.624104 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:01.624070    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm" event={"ID":"822fe93d-94ec-4d7b-96c5-de0735b61c81","Type":"ContainerStarted","Data":"94bab47f8366a81ef99c498941841928551872b94fa61e2d0b10495fffb26762"}
Jul 28 20:19:01.645078 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:01.645024    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm" podStartSLOduration=1.432555396 podStartE2EDuration="1.645007959s" totalImagesPullingTime="212.452563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:19:01.64288868 +0000 UTC m=+641.257125824" watchObservedRunningTime="2026-07-28 20:19:01.645007959 +0000 UTC m=+641.259245092"
Jul 28 20:19:10.803569 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:10.803534    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:10.803982 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:10.803596    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:10.816051 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:10.816024    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:11.683115 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:11.683087    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm"
Jul 28 20:19:14.581462 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.581424    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"]
Jul 28 20:19:14.586482 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.586463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.589088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.589067    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 28 20:19:14.596043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.596020    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"]
Jul 28 20:19:14.637014 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.636982    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:14.707059 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.707023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa44c39-d75e-40ac-ad27-262b03a2550c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.707059 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.707058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.707318 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.707094    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2k8h\" (UniqueName: \"kubernetes.io/projected/aaa44c39-d75e-40ac-ad27-262b03a2550c-kube-api-access-r2k8h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.707318 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.707171    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.707318 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.707234    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.707318 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.707289    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.808600 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.808557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa44c39-d75e-40ac-ad27-262b03a2550c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.808600 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.808602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.809103 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.808647    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2k8h\" (UniqueName: \"kubernetes.io/projected/aaa44c39-d75e-40ac-ad27-262b03a2550c-kube-api-access-r2k8h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.809103 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.808678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.809103 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.808708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.809103 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.808739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.809103 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.809088    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.809419 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.809191    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.809419 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.809219    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.810961 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.810943    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa44c39-d75e-40ac-ad27-262b03a2550c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.811213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.811196    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa44c39-d75e-40ac-ad27-262b03a2550c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.824160 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.824139    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2k8h\" (UniqueName: \"kubernetes.io/projected/aaa44c39-d75e-40ac-ad27-262b03a2550c-kube-api-access-r2k8h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v2n2g\" (UID: \"aaa44c39-d75e-40ac-ad27-262b03a2550c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:14.899108 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:14.899077    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:15.057978 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:15.057869    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"]
Jul 28 20:19:15.060519 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:19:15.060489    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa44c39_d75e_40ac_ad27_262b03a2550c.slice/crio-978c7d1949857bbbb89f6d484a96c02e480b701ceb8dcf4a3dd56a1210b5e8d9 WatchSource:0}: Error finding container 978c7d1949857bbbb89f6d484a96c02e480b701ceb8dcf4a3dd56a1210b5e8d9: Status 404 returned error can't find the container with id 978c7d1949857bbbb89f6d484a96c02e480b701ceb8dcf4a3dd56a1210b5e8d9
Jul 28 20:19:15.691066 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:15.691031    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g" event={"ID":"aaa44c39-d75e-40ac-ad27-262b03a2550c","Type":"ContainerStarted","Data":"f82fb06495280882ac788bed1276e2ab158174623c43ea14f0ef0e3c29d0014f"}
Jul 28 20:19:15.691503 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:15.691085    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g" event={"ID":"aaa44c39-d75e-40ac-ad27-262b03a2550c","Type":"ContainerStarted","Data":"978c7d1949857bbbb89f6d484a96c02e480b701ceb8dcf4a3dd56a1210b5e8d9"}
Jul 28 20:19:15.712379 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:15.712329    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g" podStartSLOduration=1.458482584 podStartE2EDuration="1.712314196s" totalImagesPullingTime="253.831612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:19:15.710542627 +0000 UTC m=+655.324779759" watchObservedRunningTime="2026-07-28 20:19:15.712314196 +0000 UTC m=+655.326551328"
Jul 28 20:19:18.635783 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:18.635727    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:19.599938 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:19.599900    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:24.899642 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:24.899598    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:24.900296 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:24.899674    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:24.912023 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:24.911999    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:19:25.750127 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:19:25.750098    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v2n2g"
Jul 28 20:20:41.521646 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:20:41.521554    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:20:41.621555 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:20:41.621508    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:20:42.745330 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:20:42.745295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:20:42.846552 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:20:42.846518    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:21:26.636359 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:26.636295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:21:26.737244 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:26.737182    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:21:31.526939 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:31.526907    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-kq6lp"]
Jul 28 20:21:31.527438 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:31.527148    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp" podUID="3e9cbe82-6bc3-4987-9bc2-bbf7111cff37" containerName="manager" containerID="cri-o://bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8" gracePeriod=30
Jul 28 20:21:31.777500 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:31.777447    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:21:31.890724 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:31.890683    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mvd9\" (UniqueName: \"kubernetes.io/projected/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-kube-api-access-5mvd9\") pod \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\" (UID: \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\") "
Jul 28 20:21:31.890926 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:31.890734    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-webhook-cert\") pod \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\" (UID: \"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37\") "
Jul 28 20:21:31.892828 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:31.892802    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3e9cbe82-6bc3-4987-9bc2-bbf7111cff37" (UID: "3e9cbe82-6bc3-4987-9bc2-bbf7111cff37"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:21:31.892928 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:31.892881    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-kube-api-access-5mvd9" (OuterVolumeSpecName: "kube-api-access-5mvd9") pod "3e9cbe82-6bc3-4987-9bc2-bbf7111cff37" (UID: "3e9cbe82-6bc3-4987-9bc2-bbf7111cff37"). InnerVolumeSpecName "kube-api-access-5mvd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:21:31.991514 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:31.991477    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mvd9\" (UniqueName: \"kubernetes.io/projected/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-kube-api-access-5mvd9\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:21:31.991514 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:31.991508    2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37-webhook-cert\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:21:32.323405 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:32.323371    2575 generic.go:356] "Generic (PLEG): container finished" podID="3e9cbe82-6bc3-4987-9bc2-bbf7111cff37" containerID="bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8" exitCode=0
Jul 28 20:21:32.323584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:32.323419    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp"
Jul 28 20:21:32.323584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:32.323458    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp" event={"ID":"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37","Type":"ContainerDied","Data":"bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8"}
Jul 28 20:21:32.323584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:32.323509    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dc5fc9c88-kq6lp" event={"ID":"3e9cbe82-6bc3-4987-9bc2-bbf7111cff37","Type":"ContainerDied","Data":"e28f1e26623fe25a2a4f930df9edb7c9012f4849a17dc96c2e5f7855a8a9ea1b"}
Jul 28 20:21:32.323584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:32.323531    2575 scope.go:117] "RemoveContainer" containerID="bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8"
Jul 28 20:21:32.333562 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:32.333544    2575 scope.go:117] "RemoveContainer" containerID="bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8"
Jul 28 20:21:32.333845 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:21:32.333823    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8\": container with ID starting with bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8 not found: ID does not exist" containerID="bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8"
Jul 28 20:21:32.333939 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:32.333851    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8"} err="failed to get container status \"bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8\": rpc error: code = NotFound desc = could not find container \"bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8\": container with ID starting with bd982cfcbc27f7fab4771a5786fc05d815ee728a3a88dc2887992dc4f803c6d8 not found: ID does not exist"
Jul 28 20:21:32.344095 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:32.344068    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-kq6lp"]
Jul 28 20:21:32.348011 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:32.347989    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-kq6lp"]
Jul 28 20:21:32.840802 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:32.840768    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e9cbe82-6bc3-4987-9bc2-bbf7111cff37" path="/var/lib/kubelet/pods/3e9cbe82-6bc3-4987-9bc2-bbf7111cff37/volumes"
Jul 28 20:21:46.458454 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:21:46.458371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:22:33.180243 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.180188    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-pmz4f"]
Jul 28 20:22:33.180737 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.180554    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9cbe82-6bc3-4987-9bc2-bbf7111cff37" containerName="manager"
Jul 28 20:22:33.180737 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.180566    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9cbe82-6bc3-4987-9bc2-bbf7111cff37" containerName="manager"
Jul 28 20:22:33.180737 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.180627    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9cbe82-6bc3-4987-9bc2-bbf7111cff37" containerName="manager"
Jul 28 20:22:33.184951 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.184932    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:33.187463 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.187437    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 20:22:33.187463 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.187454    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fbtrn\""
Jul 28 20:22:33.188498 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.188472    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 20:22:33.196132 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.196111    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-pmz4f"]
Jul 28 20:22:33.324410 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.324366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbc11275-23ae-406a-b1d6-d25dac6a94d1-webhook-cert\") pod \"maas-controller-5dc5fc9c88-pmz4f\" (UID: \"dbc11275-23ae-406a-b1d6-d25dac6a94d1\") " pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:33.324578 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.324436    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97glt\" (UniqueName: \"kubernetes.io/projected/dbc11275-23ae-406a-b1d6-d25dac6a94d1-kube-api-access-97glt\") pod \"maas-controller-5dc5fc9c88-pmz4f\" (UID: \"dbc11275-23ae-406a-b1d6-d25dac6a94d1\") " pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:33.425885 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.425841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97glt\" (UniqueName: \"kubernetes.io/projected/dbc11275-23ae-406a-b1d6-d25dac6a94d1-kube-api-access-97glt\") pod \"maas-controller-5dc5fc9c88-pmz4f\" (UID: \"dbc11275-23ae-406a-b1d6-d25dac6a94d1\") " pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:33.426095 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.425937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbc11275-23ae-406a-b1d6-d25dac6a94d1-webhook-cert\") pod \"maas-controller-5dc5fc9c88-pmz4f\" (UID: \"dbc11275-23ae-406a-b1d6-d25dac6a94d1\") " pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:33.428518 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.428489    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbc11275-23ae-406a-b1d6-d25dac6a94d1-webhook-cert\") pod \"maas-controller-5dc5fc9c88-pmz4f\" (UID: \"dbc11275-23ae-406a-b1d6-d25dac6a94d1\") " pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:33.435049 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.434991    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97glt\" (UniqueName: \"kubernetes.io/projected/dbc11275-23ae-406a-b1d6-d25dac6a94d1-kube-api-access-97glt\") pod \"maas-controller-5dc5fc9c88-pmz4f\" (UID: \"dbc11275-23ae-406a-b1d6-d25dac6a94d1\") " pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:33.495699 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.495650    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:33.634803 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:33.634770    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-pmz4f"]
Jul 28 20:22:33.637332 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:22:33.637307    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc11275_23ae_406a_b1d6_d25dac6a94d1.slice/crio-c7f1b8f5b2e6206f59270563bedb49a89ca88bb5d0f13b640798753098813cc9 WatchSource:0}: Error finding container c7f1b8f5b2e6206f59270563bedb49a89ca88bb5d0f13b640798753098813cc9: Status 404 returned error can't find the container with id c7f1b8f5b2e6206f59270563bedb49a89ca88bb5d0f13b640798753098813cc9
Jul 28 20:22:34.616260 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:34.616156    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f" event={"ID":"dbc11275-23ae-406a-b1d6-d25dac6a94d1","Type":"ContainerStarted","Data":"b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7"}
Jul 28 20:22:34.616260 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:34.616218    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f" event={"ID":"dbc11275-23ae-406a-b1d6-d25dac6a94d1","Type":"ContainerStarted","Data":"c7f1b8f5b2e6206f59270563bedb49a89ca88bb5d0f13b640798753098813cc9"}
Jul 28 20:22:34.616689 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:34.616336    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:34.637746 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:34.637694    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f" podStartSLOduration=0.974436358 podStartE2EDuration="1.637674836s" totalImagesPullingTime="663.238478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:22:34.636797754 +0000 UTC m=+854.251034889" watchObservedRunningTime="2026-07-28 20:22:34.637674836 +0000 UTC m=+854.251912034"
Jul 28 20:22:45.622815 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:45.622779    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:50.203070 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.203035    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:22:50.299195 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.299157    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-pmz4f"]
Jul 28 20:22:50.299393 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.299372    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f" podUID="dbc11275-23ae-406a-b1d6-d25dac6a94d1" containerName="manager" containerID="cri-o://b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7" gracePeriod=30
Jul 28 20:22:50.545490 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.545466    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:50.560500 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.560474    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbc11275-23ae-406a-b1d6-d25dac6a94d1-webhook-cert\") pod \"dbc11275-23ae-406a-b1d6-d25dac6a94d1\" (UID: \"dbc11275-23ae-406a-b1d6-d25dac6a94d1\") "
Jul 28 20:22:50.560634 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.560537    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97glt\" (UniqueName: \"kubernetes.io/projected/dbc11275-23ae-406a-b1d6-d25dac6a94d1-kube-api-access-97glt\") pod \"dbc11275-23ae-406a-b1d6-d25dac6a94d1\" (UID: \"dbc11275-23ae-406a-b1d6-d25dac6a94d1\") "
Jul 28 20:22:50.562633 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.562604    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc11275-23ae-406a-b1d6-d25dac6a94d1-kube-api-access-97glt" (OuterVolumeSpecName: "kube-api-access-97glt") pod "dbc11275-23ae-406a-b1d6-d25dac6a94d1" (UID: "dbc11275-23ae-406a-b1d6-d25dac6a94d1"). InnerVolumeSpecName "kube-api-access-97glt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:22:50.562633 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.562603    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc11275-23ae-406a-b1d6-d25dac6a94d1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dbc11275-23ae-406a-b1d6-d25dac6a94d1" (UID: "dbc11275-23ae-406a-b1d6-d25dac6a94d1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:22:50.661431 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.661404    2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbc11275-23ae-406a-b1d6-d25dac6a94d1-webhook-cert\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:22:50.661431 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.661428    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97glt\" (UniqueName: \"kubernetes.io/projected/dbc11275-23ae-406a-b1d6-d25dac6a94d1-kube-api-access-97glt\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:22:50.695403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.695367    2575 generic.go:356] "Generic (PLEG): container finished" podID="dbc11275-23ae-406a-b1d6-d25dac6a94d1" containerID="b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7" exitCode=0
Jul 28 20:22:50.695588 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.695412    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f"
Jul 28 20:22:50.695588 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.695455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f" event={"ID":"dbc11275-23ae-406a-b1d6-d25dac6a94d1","Type":"ContainerDied","Data":"b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7"}
Jul 28 20:22:50.695588 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.695503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dc5fc9c88-pmz4f" event={"ID":"dbc11275-23ae-406a-b1d6-d25dac6a94d1","Type":"ContainerDied","Data":"c7f1b8f5b2e6206f59270563bedb49a89ca88bb5d0f13b640798753098813cc9"}
Jul 28 20:22:50.695588 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.695524    2575 scope.go:117] "RemoveContainer" containerID="b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7"
Jul 28 20:22:50.705050 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.705029    2575 scope.go:117] "RemoveContainer" containerID="b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7"
Jul 28 20:22:50.705378 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:22:50.705350    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7\": container with ID starting with b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7 not found: ID does not exist" containerID="b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7"
Jul 28 20:22:50.705482 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.705390    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7"} err="failed to get container status \"b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7\": rpc error: code = NotFound desc = could not find container \"b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7\": container with ID starting with b7c74f116e7506a7885acbcda5b9207c71003bc1257f7790b11a98b11ea72ce7 not found: ID does not exist"
Jul 28 20:22:50.721544 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.721450    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-pmz4f"]
Jul 28 20:22:50.723946 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.723920    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-pmz4f"]
Jul 28 20:22:50.836272 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:50.836245    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc11275-23ae-406a-b1d6-d25dac6a94d1" path="/var/lib/kubelet/pods/dbc11275-23ae-406a-b1d6-d25dac6a94d1/volumes"
Jul 28 20:22:51.500016 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.499986    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:22:51.610575 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.610537    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-n2wmz"]
Jul 28 20:22:51.610917 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.610906    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc11275-23ae-406a-b1d6-d25dac6a94d1" containerName="manager"
Jul 28 20:22:51.610959 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.610919    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc11275-23ae-406a-b1d6-d25dac6a94d1" containerName="manager"
Jul 28 20:22:51.610995 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.610974    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc11275-23ae-406a-b1d6-d25dac6a94d1" containerName="manager"
Jul 28 20:22:51.615286 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.615265    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz"
Jul 28 20:22:51.619878 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.619855    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fbtrn\""
Jul 28 20:22:51.620020 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.619892    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 20:22:51.620020 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.619904    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 20:22:51.633644 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.633615    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-n2wmz"]
Jul 28 20:22:51.668652 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.668620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmpb\" (UniqueName: \"kubernetes.io/projected/9237cb59-f382-4351-b6f6-7b6b1c5cf816-kube-api-access-mjmpb\") pod \"maas-controller-5dc5fc9c88-n2wmz\" (UID: \"9237cb59-f382-4351-b6f6-7b6b1c5cf816\") " pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz"
Jul 28 20:22:51.668824 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.668660    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9237cb59-f382-4351-b6f6-7b6b1c5cf816-webhook-cert\") pod \"maas-controller-5dc5fc9c88-n2wmz\" (UID: \"9237cb59-f382-4351-b6f6-7b6b1c5cf816\") " pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz"
Jul 28 20:22:51.769713 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.769619    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmpb\" (UniqueName: \"kubernetes.io/projected/9237cb59-f382-4351-b6f6-7b6b1c5cf816-kube-api-access-mjmpb\") pod \"maas-controller-5dc5fc9c88-n2wmz\" (UID: \"9237cb59-f382-4351-b6f6-7b6b1c5cf816\") " pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz"
Jul 28 20:22:51.769713 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.769670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9237cb59-f382-4351-b6f6-7b6b1c5cf816-webhook-cert\") pod \"maas-controller-5dc5fc9c88-n2wmz\" (UID: \"9237cb59-f382-4351-b6f6-7b6b1c5cf816\") " pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz"
Jul 28 20:22:51.772134 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.772114    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9237cb59-f382-4351-b6f6-7b6b1c5cf816-webhook-cert\") pod \"maas-controller-5dc5fc9c88-n2wmz\" (UID: \"9237cb59-f382-4351-b6f6-7b6b1c5cf816\") " pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz"
Jul 28 20:22:51.780860 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.780834    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmpb\" (UniqueName: \"kubernetes.io/projected/9237cb59-f382-4351-b6f6-7b6b1c5cf816-kube-api-access-mjmpb\") pod \"maas-controller-5dc5fc9c88-n2wmz\" (UID: \"9237cb59-f382-4351-b6f6-7b6b1c5cf816\") " pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz"
Jul 28 20:22:51.925891 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:51.925847    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz"
Jul 28 20:22:52.088310 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:52.088197    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dc5fc9c88-n2wmz"]
Jul 28 20:22:52.091461 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:22:52.091431    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9237cb59_f382_4351_b6f6_7b6b1c5cf816.slice/crio-0034192cc82126a080a82a6d315241a8a32cb9532e74ff86711e17262546cd4c WatchSource:0}: Error finding container 0034192cc82126a080a82a6d315241a8a32cb9532e74ff86711e17262546cd4c: Status 404 returned error can't find the container with id 0034192cc82126a080a82a6d315241a8a32cb9532e74ff86711e17262546cd4c
Jul 28 20:22:52.714847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:52.714754    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz" event={"ID":"9237cb59-f382-4351-b6f6-7b6b1c5cf816","Type":"ContainerStarted","Data":"ebef3253393b0247d222137c0758795f9cc4e428286abd8d030d3fc60a8ac925"}
Jul 28 20:22:52.714847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:52.714805    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz" event={"ID":"9237cb59-f382-4351-b6f6-7b6b1c5cf816","Type":"ContainerStarted","Data":"0034192cc82126a080a82a6d315241a8a32cb9532e74ff86711e17262546cd4c"}
Jul 28 20:22:52.715334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:52.714905    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz"
Jul 28 20:22:52.735902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:22:52.735841    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz" podStartSLOduration=1.389684145 podStartE2EDuration="1.735819046s" totalImagesPullingTime="346.134901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:22:52.7335377 +0000 UTC m=+872.347774867" watchObservedRunningTime="2026-07-28 20:22:52.735819046 +0000 UTC m=+872.350056185"
Jul 28 20:23:03.721962 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:23:03.721931    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dc5fc9c88-n2wmz"
Jul 28 20:23:30.549707 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:23:30.549660    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:23:30.934266 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:23:30.934207    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:23:32.269581 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:23:32.269544    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:23:37.932600 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:23:37.932564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:23:38.068177 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:23:38.068120    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:23:38.247433 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:23:38.247334    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:23:38.377356 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:23:38.377319    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:23:45.091870 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:23:45.091833    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:23:45.178455 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:23:45.178420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:24:23.836270 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:24:23.836211    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:24:59.729354 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:24:59.729276    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:25:52.677537 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:25:52.677494    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:25:52.893154 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:25:52.893115    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:10.792515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:10.792472    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"]
Jul 28 20:28:10.793207 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:10.792722    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h" podUID="373dceac-889f-4689-943c-c3a13893e917" containerName="manager" containerID="cri-o://ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525" gracePeriod=10
Jul 28 20:28:13.744748 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:13.744720    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:28:13.849598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:13.849517    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/373dceac-889f-4689-943c-c3a13893e917-extensions-socket-volume\") pod \"373dceac-889f-4689-943c-c3a13893e917\" (UID: \"373dceac-889f-4689-943c-c3a13893e917\") "
Jul 28 20:28:13.849598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:13.849570    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxpvt\" (UniqueName: \"kubernetes.io/projected/373dceac-889f-4689-943c-c3a13893e917-kube-api-access-fxpvt\") pod \"373dceac-889f-4689-943c-c3a13893e917\" (UID: \"373dceac-889f-4689-943c-c3a13893e917\") "
Jul 28 20:28:13.849955 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:13.849927    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373dceac-889f-4689-943c-c3a13893e917-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "373dceac-889f-4689-943c-c3a13893e917" (UID: "373dceac-889f-4689-943c-c3a13893e917"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:28:13.851630 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:13.851611    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373dceac-889f-4689-943c-c3a13893e917-kube-api-access-fxpvt" (OuterVolumeSpecName: "kube-api-access-fxpvt") pod "373dceac-889f-4689-943c-c3a13893e917" (UID: "373dceac-889f-4689-943c-c3a13893e917"). InnerVolumeSpecName "kube-api-access-fxpvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:28:13.950149 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:13.950113    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/373dceac-889f-4689-943c-c3a13893e917-extensions-socket-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:28:13.950149 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:13.950143    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxpvt\" (UniqueName: \"kubernetes.io/projected/373dceac-889f-4689-943c-c3a13893e917-kube-api-access-fxpvt\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:28:14.195824 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:14.195796    2575 generic.go:356] "Generic (PLEG): container finished" podID="373dceac-889f-4689-943c-c3a13893e917" containerID="ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525" exitCode=0
Jul 28 20:28:14.195963 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:14.195835    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h" event={"ID":"373dceac-889f-4689-943c-c3a13893e917","Type":"ContainerDied","Data":"ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525"}
Jul 28 20:28:14.195963 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:14.195873    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h" event={"ID":"373dceac-889f-4689-943c-c3a13893e917","Type":"ContainerDied","Data":"009bf33ff5c6fdb67407d3b3ecbc8f4a4263ed299bcff90712c545ee52f3befd"}
Jul 28 20:28:14.195963 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:14.195895    2575 scope.go:117] "RemoveContainer" containerID="ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525"
Jul 28 20:28:14.195963 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:14.195837    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"
Jul 28 20:28:14.207998 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:14.207980    2575 scope.go:117] "RemoveContainer" containerID="ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525"
Jul 28 20:28:14.208276 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:28:14.208259    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525\": container with ID starting with ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525 not found: ID does not exist" containerID="ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525"
Jul 28 20:28:14.208335 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:14.208286    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525"} err="failed to get container status \"ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525\": rpc error: code = NotFound desc = could not find container \"ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525\": container with ID starting with ef3e85389a2481e69bc8dd1c1b9b68c241e31307e97efc93f8967c7d1f12e525 not found: ID does not exist"
Jul 28 20:28:14.216819 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:14.216797    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"]
Jul 28 20:28:14.223627 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:14.223608    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8np8h"]
Jul 28 20:28:14.835125 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:28:14.835095    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373dceac-889f-4689-943c-c3a13893e917" path="/var/lib/kubelet/pods/373dceac-889f-4689-943c-c3a13893e917/volumes"
Jul 28 20:29:12.011690 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.011651    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"]
Jul 28 20:29:12.014046 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.012007    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373dceac-889f-4689-943c-c3a13893e917" containerName="manager"
Jul 28 20:29:12.014046 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.012018    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="373dceac-889f-4689-943c-c3a13893e917" containerName="manager"
Jul 28 20:29:12.014046 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.012083    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="373dceac-889f-4689-943c-c3a13893e917" containerName="manager"
Jul 28 20:29:12.014921 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.014905    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"
Jul 28 20:29:12.017096 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.017077    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4qq2g\""
Jul 28 20:29:12.031745 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.031723    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"]
Jul 28 20:29:12.036059 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.036037    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/382a59c7-d2c2-475f-8dde-ded1e0004843-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qmqg5\" (UID: \"382a59c7-d2c2-475f-8dde-ded1e0004843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"
Jul 28 20:29:12.036182 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.036101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twn2w\" (UniqueName: \"kubernetes.io/projected/382a59c7-d2c2-475f-8dde-ded1e0004843-kube-api-access-twn2w\") pod \"kuadrant-operator-controller-manager-86878c769f-qmqg5\" (UID: \"382a59c7-d2c2-475f-8dde-ded1e0004843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"
Jul 28 20:29:12.137521 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.137491    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/382a59c7-d2c2-475f-8dde-ded1e0004843-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qmqg5\" (UID: \"382a59c7-d2c2-475f-8dde-ded1e0004843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"
Jul 28 20:29:12.137669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.137541    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twn2w\" (UniqueName: \"kubernetes.io/projected/382a59c7-d2c2-475f-8dde-ded1e0004843-kube-api-access-twn2w\") pod \"kuadrant-operator-controller-manager-86878c769f-qmqg5\" (UID: \"382a59c7-d2c2-475f-8dde-ded1e0004843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"
Jul 28 20:29:12.137915 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.137885    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/382a59c7-d2c2-475f-8dde-ded1e0004843-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qmqg5\" (UID: \"382a59c7-d2c2-475f-8dde-ded1e0004843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"
Jul 28 20:29:12.146656 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.146627    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twn2w\" (UniqueName: \"kubernetes.io/projected/382a59c7-d2c2-475f-8dde-ded1e0004843-kube-api-access-twn2w\") pod \"kuadrant-operator-controller-manager-86878c769f-qmqg5\" (UID: \"382a59c7-d2c2-475f-8dde-ded1e0004843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"
Jul 28 20:29:12.325140 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.325060    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"
Jul 28 20:29:12.462182 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:12.462113    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"]
Jul 28 20:29:12.465435 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:29:12.465409    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382a59c7_d2c2_475f_8dde_ded1e0004843.slice/crio-7dd597d0004b12788c0760933709d68cfe1dc5ccccbcc210439b9b3e77c3c266 WatchSource:0}: Error finding container 7dd597d0004b12788c0760933709d68cfe1dc5ccccbcc210439b9b3e77c3c266: Status 404 returned error can't find the container with id 7dd597d0004b12788c0760933709d68cfe1dc5ccccbcc210439b9b3e77c3c266
Jul 28 20:29:13.460115 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:13.460077    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5" event={"ID":"382a59c7-d2c2-475f-8dde-ded1e0004843","Type":"ContainerStarted","Data":"de2fea5c6174dfcb50cd1ee6a9598b5c5f063a14a3863cf9e3330a164ec2e980"}
Jul 28 20:29:13.460526 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:13.460132    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5" event={"ID":"382a59c7-d2c2-475f-8dde-ded1e0004843","Type":"ContainerStarted","Data":"7dd597d0004b12788c0760933709d68cfe1dc5ccccbcc210439b9b3e77c3c266"}
Jul 28 20:29:13.460526 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:13.460216    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"
Jul 28 20:29:13.484591 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:13.484544    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5" podStartSLOduration=2.484529366 podStartE2EDuration="2.484529366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:13.482460935 +0000 UTC m=+1253.096698058" watchObservedRunningTime="2026-07-28 20:29:13.484529366 +0000 UTC m=+1253.098766498"
Jul 28 20:29:24.467274 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:29:24.467218    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qmqg5"
Jul 28 20:30:21.171870 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:30:21.171839    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:40.123088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:31:40.123046    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:12.057916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.057877    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"]
Jul 28 20:32:12.061343 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.061326    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.075427 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.075402    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-35ec9396-kserve-self-signed-certs\""
Jul 28 20:32:12.154778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.154744    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:12.164887 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.164857    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"]
Jul 28 20:32:12.175749 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.175715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tmp-dir\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.175884 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.175793    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-home\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.175884 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.175875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-model-cache\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.175968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.175910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tls-certs\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.175968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.175940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rhpk\" (UniqueName: \"kubernetes.io/projected/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-kube-api-access-6rhpk\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.176040 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.175971    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-dshm\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.276836 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.276803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-model-cache\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.277030 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.276846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tls-certs\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.277030 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.276868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rhpk\" (UniqueName: \"kubernetes.io/projected/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-kube-api-access-6rhpk\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.277030 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.276891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-dshm\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.277030 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.277020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tmp-dir\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.277312 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.277083    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-home\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.277370 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.277311    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-model-cache\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.277417 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.277395    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tmp-dir\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.277462 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.277407    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-home\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.279123 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.279105    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-dshm\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.279964 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.279945    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tls-certs\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.287555 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.287529    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rhpk\" (UniqueName: \"kubernetes.io/projected/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-kube-api-access-6rhpk\") pod \"e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") " pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.370643 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.370562    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:12.523502 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:12.523475    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"]
Jul 28 20:32:12.525948 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:32:12.525921    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0034e5c7_0dd2_4aef_9ca5_1a84b28e1bf7.slice/crio-9e0cfb9c471a8435afbdeb48fa268b6d88c27385613f700fae2d94cdb2657eb6 WatchSource:0}: Error finding container 9e0cfb9c471a8435afbdeb48fa268b6d88c27385613f700fae2d94cdb2657eb6: Status 404 returned error can't find the container with id 9e0cfb9c471a8435afbdeb48fa268b6d88c27385613f700fae2d94cdb2657eb6
Jul 28 20:32:13.292388 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:13.292353    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf" event={"ID":"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7","Type":"ContainerStarted","Data":"76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f"}
Jul 28 20:32:13.292388 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:13.292394    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf" event={"ID":"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7","Type":"ContainerStarted","Data":"9e0cfb9c471a8435afbdeb48fa268b6d88c27385613f700fae2d94cdb2657eb6"}
Jul 28 20:32:13.328114 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:13.328061    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf" podStartSLOduration=1.328044246 podStartE2EDuration="1.328044246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:32:13.32504359 +0000 UTC m=+1432.939280723" watchObservedRunningTime="2026-07-28 20:32:13.328044246 +0000 UTC m=+1432.942281375"
Jul 28 20:32:18.119780 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:18.119728    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:22.371476 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:22.371389    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:22.371476 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:22.371442    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:22.383658 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:22.383631    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:23.352752 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:23.352719    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:24.724171 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:24.724117    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:39.314787 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:39.314731    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:39.756860 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:39.756830    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"]
Jul 28 20:32:39.787770 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:39.787732    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:40.684182 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:40.684147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:54.775825 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:54.775761    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf" podUID="0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" containerName="main" containerID="cri-o://76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f" gracePeriod=45
Jul 28 20:32:55.037552 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.037485    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:55.164858 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.164829    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tls-certs\") pod \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") "
Jul 28 20:32:55.165043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.164865    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-model-cache\") pod \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") "
Jul 28 20:32:55.165043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.164896    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rhpk\" (UniqueName: \"kubernetes.io/projected/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-kube-api-access-6rhpk\") pod \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") "
Jul 28 20:32:55.165043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.164953    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-dshm\") pod \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") "
Jul 28 20:32:55.165043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.164974    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tmp-dir\") pod \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") "
Jul 28 20:32:55.165043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.165020    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-home\") pod \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\" (UID: \"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7\") "
Jul 28 20:32:55.165331 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.165207    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-model-cache" (OuterVolumeSpecName: "model-cache") pod "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" (UID: "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:32:55.165331 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.165289    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" (UID: "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:32:55.165402 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.165381    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-home" (OuterVolumeSpecName: "home") pod "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" (UID: "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:32:55.166948 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.166927    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" (UID: "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:32:55.167035 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.166980    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-kube-api-access-6rhpk" (OuterVolumeSpecName: "kube-api-access-6rhpk") pod "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" (UID: "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7"). InnerVolumeSpecName "kube-api-access-6rhpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:32:55.167072 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.167032    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-dshm" (OuterVolumeSpecName: "dshm") pod "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" (UID: "0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:32:55.266635 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.266590    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tls-certs\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:32:55.266635 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.266625    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-model-cache\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:32:55.266635 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.266636    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rhpk\" (UniqueName: \"kubernetes.io/projected/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-kube-api-access-6rhpk\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:32:55.266635 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.266644    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-dshm\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:32:55.267043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.266654    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-tmp-dir\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:32:55.267043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.266666    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7-home\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:32:55.490734 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.490694    2575 generic.go:356] "Generic (PLEG): container finished" podID="0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" containerID="76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f" exitCode=0
Jul 28 20:32:55.490734 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.490736    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"
Jul 28 20:32:55.490968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.490735    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf" event={"ID":"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7","Type":"ContainerDied","Data":"76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f"}
Jul 28 20:32:55.490968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.490860    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf" event={"ID":"0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7","Type":"ContainerDied","Data":"9e0cfb9c471a8435afbdeb48fa268b6d88c27385613f700fae2d94cdb2657eb6"}
Jul 28 20:32:55.490968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.490886    2575 scope.go:117] "RemoveContainer" containerID="76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f"
Jul 28 20:32:55.500530 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.500511    2575 scope.go:117] "RemoveContainer" containerID="76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f"
Jul 28 20:32:55.500792 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:32:55.500775    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f\": container with ID starting with 76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f not found: ID does not exist" containerID="76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f"
Jul 28 20:32:55.500860 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.500799    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f"} err="failed to get container status \"76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f\": rpc error: code = NotFound desc = could not find container \"76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f\": container with ID starting with 76696b2a9587794293e037f90a3d08840d1ac1b2d6000c71cc79e3d47f08df4f not found: ID does not exist"
Jul 28 20:32:55.516876 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.516845    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"]
Jul 28 20:32:55.521357 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:55.521337    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-35ec9396-kserve-667dc4b5c5-kbznf"]
Jul 28 20:32:56.842133 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:32:56.842096    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" path="/var/lib/kubelet/pods/0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7/volumes"
Jul 28 20:33:21.485527 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:33:21.485494    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:35:30.987870 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:35:30.987790    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:35:32.287924 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:35:32.287895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:35:50.316869 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:35:50.316825    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:35:57.244624 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:35:57.244586    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:10.044313 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:10.044271    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:13.157032 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:13.156999    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:13.251638 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:13.251607    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:13.638896 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:13.638801    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:19.401031 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:19.400997    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:19.504758 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:19.504722    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:19.650597 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:19.650562    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:20.033217 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:20.033181    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:20.129486 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:20.129453    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:22.827371 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.827332    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:22.898963 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.898930    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"]
Jul 28 20:36:22.899362 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.899347    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" containerName="main"
Jul 28 20:36:22.899432 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.899365    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" containerName="main"
Jul 28 20:36:22.899486 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.899434    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0034e5c7-0dd2-4aef-9ca5-1a84b28e1bf7" containerName="main"
Jul 28 20:36:22.902589 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.902569    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:22.905122 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.905099    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d20ffe25-serving-cert\""
Jul 28 20:36:22.917360 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.917335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-openshift-service-ca\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:22.917491 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.917385    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:22.917491 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.917417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4bp7\" (UniqueName: \"kubernetes.io/projected/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-kube-api-access-j4bp7\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:22.918966 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.918941    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"]
Jul 28 20:36:22.925013 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:22.924992    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:23.010498 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.010461    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:23.018112 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.018085    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:23.018207 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.018122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4bp7\" (UniqueName: \"kubernetes.io/projected/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-kube-api-access-j4bp7\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:23.018272 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.018247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-openshift-service-ca\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:23.018272 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:36:23.018252    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-serving-cert: secret "maas-api-e2e-ait-d20ffe25-serving-cert" not found
Jul 28 20:36:23.018362 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:36:23.018349    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls podName:880060b7-12b0-4549-bc2f-7df2fa7dd5d3 nodeName:}" failed. No retries permitted until 2026-07-28 20:36:23.518329428 +0000 UTC m=+1683.132566538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls") pod "maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj" (UID: "880060b7-12b0-4549-bc2f-7df2fa7dd5d3") : secret "maas-api-e2e-ait-d20ffe25-serving-cert" not found
Jul 28 20:36:23.018888 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.018870    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-openshift-service-ca\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:23.029202 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.029179    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4bp7\" (UniqueName: \"kubernetes.io/projected/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-kube-api-access-j4bp7\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:23.520249 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.520183    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:23.522404 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.522373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:23.522532 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:36:23.522467    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-serving-cert: secret "maas-api-e2e-ait-d20ffe25-serving-cert" not found
Jul 28 20:36:23.522532 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:36:23.522526    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls podName:880060b7-12b0-4549-bc2f-7df2fa7dd5d3 nodeName:}" failed. No retries permitted until 2026-07-28 20:36:24.522511349 +0000 UTC m=+1684.136748460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls") pod "maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj" (UID: "880060b7-12b0-4549-bc2f-7df2fa7dd5d3") : secret "maas-api-e2e-ait-d20ffe25-serving-cert" not found
Jul 28 20:36:23.619835 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.619802    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:23.710308 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.710261    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"]
Jul 28 20:36:23.713782 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.713759    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:23.719414 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.719393    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d20ffe25\""
Jul 28 20:36:23.719743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.719699    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d20ffe25-dockercfg-jldcz\""
Jul 28 20:36:23.721071 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.721031    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:23.723732 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.723713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467428db-1eec-476d-8b7d-f3da997588b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b\" (UID: \"467428db-1eec-476d-8b7d-f3da997588b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:23.723846 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.723796    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4v6d\" (UniqueName: \"kubernetes.io/projected/467428db-1eec-476d-8b7d-f3da997588b5-kube-api-access-k4v6d\") pod \"payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b\" (UID: \"467428db-1eec-476d-8b7d-f3da997588b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:23.726840 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.726821    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"]
Jul 28 20:36:23.824583 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.824476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4v6d\" (UniqueName: \"kubernetes.io/projected/467428db-1eec-476d-8b7d-f3da997588b5-kube-api-access-k4v6d\") pod \"payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b\" (UID: \"467428db-1eec-476d-8b7d-f3da997588b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:23.824583 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.824571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467428db-1eec-476d-8b7d-f3da997588b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b\" (UID: \"467428db-1eec-476d-8b7d-f3da997588b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:23.825342 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.825318    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467428db-1eec-476d-8b7d-f3da997588b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b\" (UID: \"467428db-1eec-476d-8b7d-f3da997588b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:23.834218 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:23.834192    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4v6d\" (UniqueName: \"kubernetes.io/projected/467428db-1eec-476d-8b7d-f3da997588b5-kube-api-access-k4v6d\") pod \"payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b\" (UID: \"467428db-1eec-476d-8b7d-f3da997588b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:24.025206 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.025170    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:24.160672 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.160647    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"]
Jul 28 20:36:24.162306 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:36:24.162278    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467428db_1eec_476d_8b7d_f3da997588b5.slice/crio-b64bf582d19cad5c55d3506bb128b8e932a32fc096ce2573cee5a4dabef9d5b0 WatchSource:0}: Error finding container b64bf582d19cad5c55d3506bb128b8e932a32fc096ce2573cee5a4dabef9d5b0: Status 404 returned error can't find the container with id b64bf582d19cad5c55d3506bb128b8e932a32fc096ce2573cee5a4dabef9d5b0
Jul 28 20:36:24.483166 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.483134    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b" event={"ID":"467428db-1eec-476d-8b7d-f3da997588b5","Type":"ContainerStarted","Data":"8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62"}
Jul 28 20:36:24.483332 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.483189    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b" event={"ID":"467428db-1eec-476d-8b7d-f3da997588b5","Type":"ContainerStarted","Data":"b64bf582d19cad5c55d3506bb128b8e932a32fc096ce2573cee5a4dabef9d5b0"}
Jul 28 20:36:24.483332 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.483267    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:24.507921 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.507846    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b" podStartSLOduration=1.507830483 podStartE2EDuration="1.507830483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:36:24.507582878 +0000 UTC m=+1684.121820024" watchObservedRunningTime="2026-07-28 20:36:24.507830483 +0000 UTC m=+1684.122067618"
Jul 28 20:36:24.531298 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.531269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:24.533960 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.533937    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls\") pod \"maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:24.715350 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.715311    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:24.859640 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.859615    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"]
Jul 28 20:36:24.861976 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:36:24.861949    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880060b7_12b0_4549_bc2f_7df2fa7dd5d3.slice/crio-9e29b724bd517997f9b7bd7fa3155bf41c9307f29d4d75cb00c039ea6b20df2d WatchSource:0}: Error finding container 9e29b724bd517997f9b7bd7fa3155bf41c9307f29d4d75cb00c039ea6b20df2d: Status 404 returned error can't find the container with id 9e29b724bd517997f9b7bd7fa3155bf41c9307f29d4d75cb00c039ea6b20df2d
Jul 28 20:36:24.863298 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:24.863283    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:36:25.489211 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:25.489175    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj" event={"ID":"880060b7-12b0-4549-bc2f-7df2fa7dd5d3","Type":"ContainerStarted","Data":"8364b4d09e9d501ea42b22ca88666937eca9a6f32b2b3fc12d4ddc7abf6f2627"}
Jul 28 20:36:25.489424 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:25.489402    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:25.489503 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:25.489436    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj" event={"ID":"880060b7-12b0-4549-bc2f-7df2fa7dd5d3","Type":"ContainerStarted","Data":"9e29b724bd517997f9b7bd7fa3155bf41c9307f29d4d75cb00c039ea6b20df2d"}
Jul 28 20:36:25.508965 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:25.508912    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj" podStartSLOduration=2.956386115 podStartE2EDuration="3.508896532s" totalImagesPullingTime="552.510417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:36:25.506956916 +0000 UTC m=+1685.121194049" watchObservedRunningTime="2026-07-28 20:36:25.508896532 +0000 UTC m=+1685.123133665"
Jul 28 20:36:31.499629 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:31.499598    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:35.491632 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:35.491602    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:53.474358 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.474316    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"]
Jul 28 20:36:53.475494 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.475448    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj" podUID="880060b7-12b0-4549-bc2f-7df2fa7dd5d3" containerName="maas-api" containerID="cri-o://8364b4d09e9d501ea42b22ca88666937eca9a6f32b2b3fc12d4ddc7abf6f2627" gracePeriod=30
Jul 28 20:36:53.503027 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.503002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:53.538296 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.538263    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"]
Jul 28 20:36:53.538663 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.538531    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b" podUID="467428db-1eec-476d-8b7d-f3da997588b5" containerName="payload-pre-processing" containerID="cri-o://8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62" gracePeriod=30
Jul 28 20:36:53.580081 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.580048    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:53.618573 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.618547    2575 generic.go:356] "Generic (PLEG): container finished" podID="880060b7-12b0-4549-bc2f-7df2fa7dd5d3" containerID="8364b4d09e9d501ea42b22ca88666937eca9a6f32b2b3fc12d4ddc7abf6f2627" exitCode=0
Jul 28 20:36:53.618718 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.618594    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj" event={"ID":"880060b7-12b0-4549-bc2f-7df2fa7dd5d3","Type":"ContainerDied","Data":"8364b4d09e9d501ea42b22ca88666937eca9a6f32b2b3fc12d4ddc7abf6f2627"}
Jul 28 20:36:53.681901 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.681610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:53.764158 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.764083    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:53.790892 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.790869    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:53.793818 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.793799    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:53.888770 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.888743    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-openshift-service-ca\") pod \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") "
Jul 28 20:36:53.888948 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.888856    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467428db-1eec-476d-8b7d-f3da997588b5-plugins-config-volume\") pod \"467428db-1eec-476d-8b7d-f3da997588b5\" (UID: \"467428db-1eec-476d-8b7d-f3da997588b5\") "
Jul 28 20:36:53.888948 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.888892    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4bp7\" (UniqueName: \"kubernetes.io/projected/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-kube-api-access-j4bp7\") pod \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") "
Jul 28 20:36:53.888948 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.888925    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4v6d\" (UniqueName: \"kubernetes.io/projected/467428db-1eec-476d-8b7d-f3da997588b5-kube-api-access-k4v6d\") pod \"467428db-1eec-476d-8b7d-f3da997588b5\" (UID: \"467428db-1eec-476d-8b7d-f3da997588b5\") "
Jul 28 20:36:53.889122 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.888950    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls\") pod \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\" (UID: \"880060b7-12b0-4549-bc2f-7df2fa7dd5d3\") "
Jul 28 20:36:53.889179 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.889132    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "880060b7-12b0-4549-bc2f-7df2fa7dd5d3" (UID: "880060b7-12b0-4549-bc2f-7df2fa7dd5d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:36:53.889331 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.889304    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467428db-1eec-476d-8b7d-f3da997588b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "467428db-1eec-476d-8b7d-f3da997588b5" (UID: "467428db-1eec-476d-8b7d-f3da997588b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:36:53.891109 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.891079    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467428db-1eec-476d-8b7d-f3da997588b5-kube-api-access-k4v6d" (OuterVolumeSpecName: "kube-api-access-k4v6d") pod "467428db-1eec-476d-8b7d-f3da997588b5" (UID: "467428db-1eec-476d-8b7d-f3da997588b5"). InnerVolumeSpecName "kube-api-access-k4v6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:36:53.891222 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.891130    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-kube-api-access-j4bp7" (OuterVolumeSpecName: "kube-api-access-j4bp7") pod "880060b7-12b0-4549-bc2f-7df2fa7dd5d3" (UID: "880060b7-12b0-4549-bc2f-7df2fa7dd5d3"). InnerVolumeSpecName "kube-api-access-j4bp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:36:53.891603 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.891584    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "880060b7-12b0-4549-bc2f-7df2fa7dd5d3" (UID: "880060b7-12b0-4549-bc2f-7df2fa7dd5d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:36:53.908825 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.908804    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:53.990283 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.990222    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467428db-1eec-476d-8b7d-f3da997588b5-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:36:53.990283 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.990279    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4bp7\" (UniqueName: \"kubernetes.io/projected/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-kube-api-access-j4bp7\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:36:53.990478 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.990295    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4v6d\" (UniqueName: \"kubernetes.io/projected/467428db-1eec-476d-8b7d-f3da997588b5-kube-api-access-k4v6d\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:36:53.990478 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.990308    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:36:53.990478 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:53.990320    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880060b7-12b0-4549-bc2f-7df2fa7dd5d3-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:36:54.133264 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.133158    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:54.378552 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.378515    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:54.491083 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.491055    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:54.610353 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.610311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:54.624841 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.624817    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"
Jul 28 20:36:54.624995 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.624822    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj" event={"ID":"880060b7-12b0-4549-bc2f-7df2fa7dd5d3","Type":"ContainerDied","Data":"9e29b724bd517997f9b7bd7fa3155bf41c9307f29d4d75cb00c039ea6b20df2d"}
Jul 28 20:36:54.625061 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.624968    2575 scope.go:117] "RemoveContainer" containerID="8364b4d09e9d501ea42b22ca88666937eca9a6f32b2b3fc12d4ddc7abf6f2627"
Jul 28 20:36:54.626330 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.626173    2575 generic.go:356] "Generic (PLEG): container finished" podID="467428db-1eec-476d-8b7d-f3da997588b5" containerID="8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62" exitCode=0
Jul 28 20:36:54.626330 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.626213    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"
Jul 28 20:36:54.626330 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.626269    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b" event={"ID":"467428db-1eec-476d-8b7d-f3da997588b5","Type":"ContainerDied","Data":"8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62"}
Jul 28 20:36:54.626330 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.626321    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b" event={"ID":"467428db-1eec-476d-8b7d-f3da997588b5","Type":"ContainerDied","Data":"b64bf582d19cad5c55d3506bb128b8e932a32fc096ce2573cee5a4dabef9d5b0"}
Jul 28 20:36:54.635418 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.635390    2575 scope.go:117] "RemoveContainer" containerID="8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62"
Jul 28 20:36:54.644553 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.644536    2575 scope.go:117] "RemoveContainer" containerID="8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62"
Jul 28 20:36:54.644824 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:36:54.644804    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62\": container with ID starting with 8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62 not found: ID does not exist" containerID="8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62"
Jul 28 20:36:54.644867 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.644831    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62"} err="failed to get container status \"8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62\": rpc error: code = NotFound desc = could not find container \"8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62\": container with ID starting with 8db6798e1527c88f8d1e9788f10e25dc0bb4ae110bb9890c5583aeee95f35e62 not found: ID does not exist"
Jul 28 20:36:54.653878 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.653858    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"]
Jul 28 20:36:54.663995 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.663976    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d20ffe25-7d5b47fccb-pxmsj"]
Jul 28 20:36:54.696187 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.696160    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"]
Jul 28 20:36:54.703366 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.703334    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d20ffe25-5f7fdf76f6-n787b"]
Jul 28 20:36:54.836952 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.836867    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="467428db-1eec-476d-8b7d-f3da997588b5" path="/var/lib/kubelet/pods/467428db-1eec-476d-8b7d-f3da997588b5/volumes"
Jul 28 20:36:54.837444 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.837425    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="880060b7-12b0-4549-bc2f-7df2fa7dd5d3" path="/var/lib/kubelet/pods/880060b7-12b0-4549-bc2f-7df2fa7dd5d3/volumes"
Jul 28 20:36:54.989729 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:54.989693    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:55.415747 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:55.415710    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:57.229646 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.229586    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:57.329869 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.329820    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"]
Jul 28 20:36:57.330377 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.330356    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="880060b7-12b0-4549-bc2f-7df2fa7dd5d3" containerName="maas-api"
Jul 28 20:36:57.330377 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.330378    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="880060b7-12b0-4549-bc2f-7df2fa7dd5d3" containerName="maas-api"
Jul 28 20:36:57.330560 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.330404    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467428db-1eec-476d-8b7d-f3da997588b5" containerName="payload-pre-processing"
Jul 28 20:36:57.330560 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.330414    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="467428db-1eec-476d-8b7d-f3da997588b5" containerName="payload-pre-processing"
Jul 28 20:36:57.330560 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.330510    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="880060b7-12b0-4549-bc2f-7df2fa7dd5d3" containerName="maas-api"
Jul 28 20:36:57.330560 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.330522    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="467428db-1eec-476d-8b7d-f3da997588b5" containerName="payload-pre-processing"
Jul 28 20:36:57.334901 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.334880    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.337394 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.337373    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4b47811d-serving-cert\""
Jul 28 20:36:57.343704 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.343681    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"]
Jul 28 20:36:57.374324 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.374295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:57.418368 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.418343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2jn7\" (UniqueName: \"kubernetes.io/projected/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-kube-api-access-h2jn7\") pod \"maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.418478 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.418383    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-maas-api-tls\") pod \"maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.418478 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.418458    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.502687 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.502615    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:57.519204 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.519179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.519295 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.519279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2jn7\" (UniqueName: \"kubernetes.io/projected/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-kube-api-access-h2jn7\") pod \"maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.519345 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.519306    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-maas-api-tls\") pod \"maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.519766 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.519744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.521651 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.521632    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-maas-api-tls\") pod \"maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.529688 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.529660    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2jn7\" (UniqueName: \"kubernetes.io/projected/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-kube-api-access-h2jn7\") pod \"maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.646364 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.646333    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:57.796474 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.796292    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"]
Jul 28 20:36:57.799215 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:36:57.799189    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f06d22b_a0a9_49d6_82b0_b7e9a9c32ed1.slice/crio-9d833ecf36bbeda42a75857b945279a67bdea724d33cd6588fb430ee1039cca4 WatchSource:0}: Error finding container 9d833ecf36bbeda42a75857b945279a67bdea724d33cd6588fb430ee1039cca4: Status 404 returned error can't find the container with id 9d833ecf36bbeda42a75857b945279a67bdea724d33cd6588fb430ee1039cca4
Jul 28 20:36:57.888456 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.888384    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:57.981436 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.981403    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:36:57.994731 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.994708    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"]
Jul 28 20:36:57.999167 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:57.999149    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:36:58.001939 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.001918    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4b47811d\""
Jul 28 20:36:58.002044 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.002012    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4b47811d-dockercfg-br85j\""
Jul 28 20:36:58.007103 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.007081    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"]
Jul 28 20:36:58.124760 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.124684    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bd560c-a90d-4da3-ab04-252759fa4943-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8\" (UID: \"70bd560c-a90d-4da3-ab04-252759fa4943\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:36:58.124923 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.124835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rs5r\" (UniqueName: \"kubernetes.io/projected/70bd560c-a90d-4da3-ab04-252759fa4943-kube-api-access-2rs5r\") pod \"payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8\" (UID: \"70bd560c-a90d-4da3-ab04-252759fa4943\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:36:58.226169 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.226144    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bd560c-a90d-4da3-ab04-252759fa4943-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8\" (UID: \"70bd560c-a90d-4da3-ab04-252759fa4943\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:36:58.226312 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.226198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rs5r\" (UniqueName: \"kubernetes.io/projected/70bd560c-a90d-4da3-ab04-252759fa4943-kube-api-access-2rs5r\") pod \"payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8\" (UID: \"70bd560c-a90d-4da3-ab04-252759fa4943\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:36:58.226816 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.226796    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bd560c-a90d-4da3-ab04-252759fa4943-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8\" (UID: \"70bd560c-a90d-4da3-ab04-252759fa4943\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:36:58.234182 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.234160    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rs5r\" (UniqueName: \"kubernetes.io/projected/70bd560c-a90d-4da3-ab04-252759fa4943-kube-api-access-2rs5r\") pod \"payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8\" (UID: \"70bd560c-a90d-4da3-ab04-252759fa4943\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:36:58.310792 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.310759    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:36:58.648561 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.648479    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m" event={"ID":"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1","Type":"ContainerStarted","Data":"e466b27adb031f681bf4aac6827e3d884e890d19d8689d3d14287f329aba90b2"}
Jul 28 20:36:58.648561 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.648538    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m" event={"ID":"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1","Type":"ContainerStarted","Data":"9d833ecf36bbeda42a75857b945279a67bdea724d33cd6588fb430ee1039cca4"}
Jul 28 20:36:58.648733 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.648632    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:36:58.657388 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.657362    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"]
Jul 28 20:36:58.658862 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:36:58.658840    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bd560c_a90d_4da3_ab04_252759fa4943.slice/crio-3496c18f73f412b1ec2cf652b91b298cf796acc5f8a10c48fc0004a0d4c70972 WatchSource:0}: Error finding container 3496c18f73f412b1ec2cf652b91b298cf796acc5f8a10c48fc0004a0d4c70972: Status 404 returned error can't find the container with id 3496c18f73f412b1ec2cf652b91b298cf796acc5f8a10c48fc0004a0d4c70972
Jul 28 20:36:58.667942 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:58.667893    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m" podStartSLOduration=1.3223595989999999 podStartE2EDuration="1.667874062s" totalImagesPullingTime="345.514463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:36:58.666954292 +0000 UTC m=+1718.281191426" watchObservedRunningTime="2026-07-28 20:36:58.667874062 +0000 UTC m=+1718.282111198"
Jul 28 20:36:59.654298 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:59.654263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8" event={"ID":"70bd560c-a90d-4da3-ab04-252759fa4943","Type":"ContainerStarted","Data":"ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a"}
Jul 28 20:36:59.654689 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:59.654323    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8" event={"ID":"70bd560c-a90d-4da3-ab04-252759fa4943","Type":"ContainerStarted","Data":"3496c18f73f412b1ec2cf652b91b298cf796acc5f8a10c48fc0004a0d4c70972"}
Jul 28 20:36:59.671305 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:36:59.671261    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8" podStartSLOduration=2.671246146 podStartE2EDuration="2.671246146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:36:59.669807793 +0000 UTC m=+1719.284044937" watchObservedRunningTime="2026-07-28 20:36:59.671246146 +0000 UTC m=+1719.285483314"
Jul 28 20:37:00.659190 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:00.659162    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:37:04.660442 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:04.660416    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:37:11.665766 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:11.665736    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:37:28.667583 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.667547    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"]
Jul 28 20:37:28.668106 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.667773    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m" podUID="4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1" containerName="maas-api" containerID="cri-o://e466b27adb031f681bf4aac6827e3d884e890d19d8689d3d14287f329aba90b2" gracePeriod=30
Jul 28 20:37:28.688026 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.688000    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:28.731683 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.731650    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"]
Jul 28 20:37:28.732150 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.732095    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8" podUID="70bd560c-a90d-4da3-ab04-252759fa4943" containerName="payload-pre-processing" containerID="cri-o://ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a" gracePeriod=30
Jul 28 20:37:28.785405 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.785376    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:28.798210 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.798184    2575 generic.go:356] "Generic (PLEG): container finished" podID="4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1" containerID="e466b27adb031f681bf4aac6827e3d884e890d19d8689d3d14287f329aba90b2" exitCode=0
Jul 28 20:37:28.798342 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.798260    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m" event={"ID":"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1","Type":"ContainerDied","Data":"e466b27adb031f681bf4aac6827e3d884e890d19d8689d3d14287f329aba90b2"}
Jul 28 20:37:28.864293 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.864263    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:28.925389 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.925304    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:28.974396 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:28.974341    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:37:29.016907 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.016874    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:37:29.029342 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.029314    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:29.082238 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.082208    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bd560c-a90d-4da3-ab04-252759fa4943-plugins-config-volume\") pod \"70bd560c-a90d-4da3-ab04-252759fa4943\" (UID: \"70bd560c-a90d-4da3-ab04-252759fa4943\") "
Jul 28 20:37:29.082383 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.082276    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-openshift-service-ca\") pod \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") "
Jul 28 20:37:29.082383 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.082300    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2jn7\" (UniqueName: \"kubernetes.io/projected/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-kube-api-access-h2jn7\") pod \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") "
Jul 28 20:37:29.082383 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.082320    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-maas-api-tls\") pod \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\" (UID: \"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1\") "
Jul 28 20:37:29.082383 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.082377    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rs5r\" (UniqueName: \"kubernetes.io/projected/70bd560c-a90d-4da3-ab04-252759fa4943-kube-api-access-2rs5r\") pod \"70bd560c-a90d-4da3-ab04-252759fa4943\" (UID: \"70bd560c-a90d-4da3-ab04-252759fa4943\") "
Jul 28 20:37:29.082633 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.082609    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bd560c-a90d-4da3-ab04-252759fa4943-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70bd560c-a90d-4da3-ab04-252759fa4943" (UID: "70bd560c-a90d-4da3-ab04-252759fa4943"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:37:29.082710 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.082683    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1" (UID: "4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:37:29.084539 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.084507    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bd560c-a90d-4da3-ab04-252759fa4943-kube-api-access-2rs5r" (OuterVolumeSpecName: "kube-api-access-2rs5r") pod "70bd560c-a90d-4da3-ab04-252759fa4943" (UID: "70bd560c-a90d-4da3-ab04-252759fa4943"). InnerVolumeSpecName "kube-api-access-2rs5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:37:29.084539 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.084522    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1" (UID: "4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:37:29.084539 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.084540    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-kube-api-access-h2jn7" (OuterVolumeSpecName: "kube-api-access-h2jn7") pod "4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1" (UID: "4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1"). InnerVolumeSpecName "kube-api-access-h2jn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:37:29.183860 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.183776    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rs5r\" (UniqueName: \"kubernetes.io/projected/70bd560c-a90d-4da3-ab04-252759fa4943-kube-api-access-2rs5r\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:37:29.183860 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.183803    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bd560c-a90d-4da3-ab04-252759fa4943-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:37:29.183860 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.183819    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:37:29.183860 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.183832    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2jn7\" (UniqueName: \"kubernetes.io/projected/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-kube-api-access-h2jn7\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:37:29.183860 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.183845    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:37:29.264028 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.263998    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:29.496091 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.496009    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:29.806825 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.806744    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"
Jul 28 20:37:29.806825 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.806770    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m" event={"ID":"4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1","Type":"ContainerDied","Data":"9d833ecf36bbeda42a75857b945279a67bdea724d33cd6588fb430ee1039cca4"}
Jul 28 20:37:29.807373 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.806839    2575 scope.go:117] "RemoveContainer" containerID="e466b27adb031f681bf4aac6827e3d884e890d19d8689d3d14287f329aba90b2"
Jul 28 20:37:29.809368 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.809334    2575 generic.go:356] "Generic (PLEG): container finished" podID="70bd560c-a90d-4da3-ab04-252759fa4943" containerID="ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a" exitCode=0
Jul 28 20:37:29.809478 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.809409    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"
Jul 28 20:37:29.809478 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.809438    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8" event={"ID":"70bd560c-a90d-4da3-ab04-252759fa4943","Type":"ContainerDied","Data":"ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a"}
Jul 28 20:37:29.809578 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.809504    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8" event={"ID":"70bd560c-a90d-4da3-ab04-252759fa4943","Type":"ContainerDied","Data":"3496c18f73f412b1ec2cf652b91b298cf796acc5f8a10c48fc0004a0d4c70972"}
Jul 28 20:37:29.816221 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.816204    2575 scope.go:117] "RemoveContainer" containerID="ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a"
Jul 28 20:37:29.824681 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.824664    2575 scope.go:117] "RemoveContainer" containerID="ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a"
Jul 28 20:37:29.824933 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:37:29.824913    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a\": container with ID starting with ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a not found: ID does not exist" containerID="ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a"
Jul 28 20:37:29.824984 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.824946    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a"} err="failed to get container status \"ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a\": rpc error: code = NotFound desc = could not find container \"ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a\": container with ID starting with ee9a587aa3fd47f8e912dd858e5a7d7ebc89efa4632dbe43afed9cc006dadd9a not found: ID does not exist"
Jul 28 20:37:29.840070 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.840047    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"]
Jul 28 20:37:29.849298 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.849268    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:29.859792 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.854127    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b47811d-769546d8d8-9nnq8"]
Jul 28 20:37:29.866050 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.866022    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"]
Jul 28 20:37:29.872263 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:29.872241    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b47811d-74fbd9f6cd-mhg8m"]
Jul 28 20:37:30.476296 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:30.476244    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:30.837011 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:30.836940    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1" path="/var/lib/kubelet/pods/4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1/volumes"
Jul 28 20:37:30.837410 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:30.837297    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70bd560c-a90d-4da3-ab04-252759fa4943" path="/var/lib/kubelet/pods/70bd560c-a90d-4da3-ab04-252759fa4943/volumes"
Jul 28 20:37:32.203295 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.203243    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:32.287847 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.287816    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:32.419909 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.419876    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"]
Jul 28 20:37:32.420344 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.420325    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bd560c-a90d-4da3-ab04-252759fa4943" containerName="payload-pre-processing"
Jul 28 20:37:32.420448 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.420345    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bd560c-a90d-4da3-ab04-252759fa4943" containerName="payload-pre-processing"
Jul 28 20:37:32.420448 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.420365    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1" containerName="maas-api"
Jul 28 20:37:32.420448 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.420373    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1" containerName="maas-api"
Jul 28 20:37:32.420603 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.420462    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f06d22b-a0a9-49d6-82b0-b7e9a9c32ed1" containerName="maas-api"
Jul 28 20:37:32.420603 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.420476    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="70bd560c-a90d-4da3-ab04-252759fa4943" containerName="payload-pre-processing"
Jul 28 20:37:32.424799 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.424778    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.427306 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.427284    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ed417440-serving-cert\""
Jul 28 20:37:32.438173 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.438144    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"]
Jul 28 20:37:32.461609 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.461521    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:32.512790 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.512766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwrl\" (UniqueName: \"kubernetes.io/projected/ece2e62c-c433-4c98-8646-229cbec7abe5-kube-api-access-gcwrl\") pod \"maas-api-e2e-ait-ed417440-55fc74d689-72lnm\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.512895 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.512802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece2e62c-c433-4c98-8646-229cbec7abe5-maas-api-tls\") pod \"maas-api-e2e-ait-ed417440-55fc74d689-72lnm\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.512962 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.512915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece2e62c-c433-4c98-8646-229cbec7abe5-openshift-service-ca\") pod \"maas-api-e2e-ait-ed417440-55fc74d689-72lnm\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.613318 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.613284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece2e62c-c433-4c98-8646-229cbec7abe5-maas-api-tls\") pod \"maas-api-e2e-ait-ed417440-55fc74d689-72lnm\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.613482 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.613380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece2e62c-c433-4c98-8646-229cbec7abe5-openshift-service-ca\") pod \"maas-api-e2e-ait-ed417440-55fc74d689-72lnm\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.613482 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.613407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwrl\" (UniqueName: \"kubernetes.io/projected/ece2e62c-c433-4c98-8646-229cbec7abe5-kube-api-access-gcwrl\") pod \"maas-api-e2e-ait-ed417440-55fc74d689-72lnm\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.613992 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.613969    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece2e62c-c433-4c98-8646-229cbec7abe5-openshift-service-ca\") pod \"maas-api-e2e-ait-ed417440-55fc74d689-72lnm\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.615701 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.615681    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece2e62c-c433-4c98-8646-229cbec7abe5-maas-api-tls\") pod \"maas-api-e2e-ait-ed417440-55fc74d689-72lnm\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.623411 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.623391    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwrl\" (UniqueName: \"kubernetes.io/projected/ece2e62c-c433-4c98-8646-229cbec7abe5-kube-api-access-gcwrl\") pod \"maas-api-e2e-ait-ed417440-55fc74d689-72lnm\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.649530 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.649507    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:32.736472 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.736408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:32.877832 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:32.877803    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"]
Jul 28 20:37:32.880347 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:37:32.880314    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece2e62c_c433_4c98_8646_229cbec7abe5.slice/crio-64c61ff320a11ccc7646f34aef1f3bbcb4ada479c5f8beb8e56f51e9f4d75836 WatchSource:0}: Error finding container 64c61ff320a11ccc7646f34aef1f3bbcb4ada479c5f8beb8e56f51e9f4d75836: Status 404 returned error can't find the container with id 64c61ff320a11ccc7646f34aef1f3bbcb4ada479c5f8beb8e56f51e9f4d75836
Jul 28 20:37:33.041611 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.041527    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:33.145653 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.145624    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"]
Jul 28 20:37:33.145807 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.145623    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:37:33.150076 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.150060    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:37:33.152396 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.152375    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ed417440\""
Jul 28 20:37:33.152492 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.152399    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ed417440-dockercfg-2sm2l\""
Jul 28 20:37:33.159338 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.159313    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"]
Jul 28 20:37:33.219406 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.219376    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlggj\" (UniqueName: \"kubernetes.io/projected/0ad8719c-b465-4f07-877e-892a00f20430-kube-api-access-xlggj\") pod \"payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b\" (UID: \"0ad8719c-b465-4f07-877e-892a00f20430\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:37:33.219739 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.219432    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad8719c-b465-4f07-877e-892a00f20430-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b\" (UID: \"0ad8719c-b465-4f07-877e-892a00f20430\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:37:33.320581 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.320501    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad8719c-b465-4f07-877e-892a00f20430-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b\" (UID: \"0ad8719c-b465-4f07-877e-892a00f20430\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:37:33.320581 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.320573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlggj\" (UniqueName: \"kubernetes.io/projected/0ad8719c-b465-4f07-877e-892a00f20430-kube-api-access-xlggj\") pod \"payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b\" (UID: \"0ad8719c-b465-4f07-877e-892a00f20430\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:37:33.321332 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.321312    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad8719c-b465-4f07-877e-892a00f20430-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b\" (UID: \"0ad8719c-b465-4f07-877e-892a00f20430\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:37:33.330439 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.330406    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlggj\" (UniqueName: \"kubernetes.io/projected/0ad8719c-b465-4f07-877e-892a00f20430-kube-api-access-xlggj\") pod \"payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b\" (UID: \"0ad8719c-b465-4f07-877e-892a00f20430\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:37:33.461861 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.461823    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:37:33.595591 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.595404    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"]
Jul 28 20:37:33.598071 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:37:33.598045    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad8719c_b465_4f07_877e_892a00f20430.slice/crio-8867afd58f8f5340d4917c0e1acb2da2381f48f8d59e64e57c4b6abb62525549 WatchSource:0}: Error finding container 8867afd58f8f5340d4917c0e1acb2da2381f48f8d59e64e57c4b6abb62525549: Status 404 returned error can't find the container with id 8867afd58f8f5340d4917c0e1acb2da2381f48f8d59e64e57c4b6abb62525549
Jul 28 20:37:33.836193 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.836046    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b" event={"ID":"0ad8719c-b465-4f07-877e-892a00f20430","Type":"ContainerStarted","Data":"63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38"}
Jul 28 20:37:33.836193 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.836106    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b" event={"ID":"0ad8719c-b465-4f07-877e-892a00f20430","Type":"ContainerStarted","Data":"8867afd58f8f5340d4917c0e1acb2da2381f48f8d59e64e57c4b6abb62525549"}
Jul 28 20:37:33.836455 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.836302    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:37:33.837565 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.837539    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm" event={"ID":"ece2e62c-c433-4c98-8646-229cbec7abe5","Type":"ContainerStarted","Data":"64c61ff320a11ccc7646f34aef1f3bbcb4ada479c5f8beb8e56f51e9f4d75836"}
Jul 28 20:37:33.854088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:33.854002    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b" podStartSLOduration=0.853986009 podStartE2EDuration="853.986009ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:37:33.85258192 +0000 UTC m=+1753.466819056" watchObservedRunningTime="2026-07-28 20:37:33.853986009 +0000 UTC m=+1753.468223173"
Jul 28 20:37:34.843917 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:34.843877    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm" event={"ID":"ece2e62c-c433-4c98-8646-229cbec7abe5","Type":"ContainerStarted","Data":"ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b"}
Jul 28 20:37:34.844356 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:34.844123    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:34.861109 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:34.861063    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm" podStartSLOduration=0.982310475 podStartE2EDuration="2.861048108s" totalImagesPullingTime="1.878737633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:37:34.858830262 +0000 UTC m=+1754.473067395" watchObservedRunningTime="2026-07-28 20:37:34.861048108 +0000 UTC m=+1754.475285242"
Jul 28 20:37:40.854477 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:40.854444    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:37:44.846410 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:37:44.846379    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:38:02.996519 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:02.996480    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"]
Jul 28 20:38:02.997072 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:02.996687    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm" podUID="ece2e62c-c433-4c98-8646-229cbec7abe5" containerName="maas-api" containerID="cri-o://ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b" gracePeriod=30
Jul 28 20:38:03.015382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.015351    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:03.058533 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.058496    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"]
Jul 28 20:38:03.058769 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.058744    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b" podUID="0ad8719c-b465-4f07-877e-892a00f20430" containerName="payload-pre-processing" containerID="cri-o://63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38" gracePeriod=30
Jul 28 20:38:03.116426 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.116353    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:03.211034 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.210994    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:03.282388 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.282326    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:03.306700 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.306679    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:38:03.338551 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.338525    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:38:03.374119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.374094    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlggj\" (UniqueName: \"kubernetes.io/projected/0ad8719c-b465-4f07-877e-892a00f20430-kube-api-access-xlggj\") pod \"0ad8719c-b465-4f07-877e-892a00f20430\" (UID: \"0ad8719c-b465-4f07-877e-892a00f20430\") "
Jul 28 20:38:03.374294 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.374141    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad8719c-b465-4f07-877e-892a00f20430-plugins-config-volume\") pod \"0ad8719c-b465-4f07-877e-892a00f20430\" (UID: \"0ad8719c-b465-4f07-877e-892a00f20430\") "
Jul 28 20:38:03.374294 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.374176    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece2e62c-c433-4c98-8646-229cbec7abe5-openshift-service-ca\") pod \"ece2e62c-c433-4c98-8646-229cbec7abe5\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") "
Jul 28 20:38:03.374294 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.374207    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece2e62c-c433-4c98-8646-229cbec7abe5-maas-api-tls\") pod \"ece2e62c-c433-4c98-8646-229cbec7abe5\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") "
Jul 28 20:38:03.374294 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.374265    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcwrl\" (UniqueName: \"kubernetes.io/projected/ece2e62c-c433-4c98-8646-229cbec7abe5-kube-api-access-gcwrl\") pod \"ece2e62c-c433-4c98-8646-229cbec7abe5\" (UID: \"ece2e62c-c433-4c98-8646-229cbec7abe5\") "
Jul 28 20:38:03.374577 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.374549    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece2e62c-c433-4c98-8646-229cbec7abe5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ece2e62c-c433-4c98-8646-229cbec7abe5" (UID: "ece2e62c-c433-4c98-8646-229cbec7abe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:38:03.374577 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.374561    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad8719c-b465-4f07-877e-892a00f20430-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ad8719c-b465-4f07-877e-892a00f20430" (UID: "0ad8719c-b465-4f07-877e-892a00f20430"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:38:03.376197 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.376162    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad8719c-b465-4f07-877e-892a00f20430-kube-api-access-xlggj" (OuterVolumeSpecName: "kube-api-access-xlggj") pod "0ad8719c-b465-4f07-877e-892a00f20430" (UID: "0ad8719c-b465-4f07-877e-892a00f20430"). InnerVolumeSpecName "kube-api-access-xlggj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:38:03.376375 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.376202    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece2e62c-c433-4c98-8646-229cbec7abe5-kube-api-access-gcwrl" (OuterVolumeSpecName: "kube-api-access-gcwrl") pod "ece2e62c-c433-4c98-8646-229cbec7abe5" (UID: "ece2e62c-c433-4c98-8646-229cbec7abe5"). InnerVolumeSpecName "kube-api-access-gcwrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:38:03.376375 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.376210    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece2e62c-c433-4c98-8646-229cbec7abe5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ece2e62c-c433-4c98-8646-229cbec7abe5" (UID: "ece2e62c-c433-4c98-8646-229cbec7abe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:38:03.475715 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.475686    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcwrl\" (UniqueName: \"kubernetes.io/projected/ece2e62c-c433-4c98-8646-229cbec7abe5-kube-api-access-gcwrl\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:38:03.475715 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.475714    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlggj\" (UniqueName: \"kubernetes.io/projected/0ad8719c-b465-4f07-877e-892a00f20430-kube-api-access-xlggj\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:38:03.475870 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.475725    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad8719c-b465-4f07-877e-892a00f20430-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:38:03.475870 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.475733    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece2e62c-c433-4c98-8646-229cbec7abe5-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:38:03.475870 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.475743    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece2e62c-c433-4c98-8646-229cbec7abe5-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:38:03.977219 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.977186    2575 generic.go:356] "Generic (PLEG): container finished" podID="0ad8719c-b465-4f07-877e-892a00f20430" containerID="63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38" exitCode=0
Jul 28 20:38:03.977426 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.977240    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b" event={"ID":"0ad8719c-b465-4f07-877e-892a00f20430","Type":"ContainerDied","Data":"63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38"}
Jul 28 20:38:03.977426 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.977259    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"
Jul 28 20:38:03.977426 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.977300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b" event={"ID":"0ad8719c-b465-4f07-877e-892a00f20430","Type":"ContainerDied","Data":"8867afd58f8f5340d4917c0e1acb2da2381f48f8d59e64e57c4b6abb62525549"}
Jul 28 20:38:03.977426 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.977332    2575 scope.go:117] "RemoveContainer" containerID="63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38"
Jul 28 20:38:03.978731 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.978707    2575 generic.go:356] "Generic (PLEG): container finished" podID="ece2e62c-c433-4c98-8646-229cbec7abe5" containerID="ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b" exitCode=0
Jul 28 20:38:03.978858 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.978757    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"
Jul 28 20:38:03.978858 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.978759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm" event={"ID":"ece2e62c-c433-4c98-8646-229cbec7abe5","Type":"ContainerDied","Data":"ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b"}
Jul 28 20:38:03.978976 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.978861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm" event={"ID":"ece2e62c-c433-4c98-8646-229cbec7abe5","Type":"ContainerDied","Data":"64c61ff320a11ccc7646f34aef1f3bbcb4ada479c5f8beb8e56f51e9f4d75836"}
Jul 28 20:38:03.986928 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.986910    2575 scope.go:117] "RemoveContainer" containerID="63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38"
Jul 28 20:38:03.987189 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:38:03.987166    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38\": container with ID starting with 63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38 not found: ID does not exist" containerID="63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38"
Jul 28 20:38:03.987299 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.987196    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38"} err="failed to get container status \"63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38\": rpc error: code = NotFound desc = could not find container \"63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38\": container with ID starting with 63988183fe375dd727a78fc58637aef184cf8c1b08a5efa4ad7f6d283377bb38 not found: ID does not exist"
Jul 28 20:38:03.987299 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.987212    2575 scope.go:117] "RemoveContainer" containerID="ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b"
Jul 28 20:38:03.994690 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.994674    2575 scope.go:117] "RemoveContainer" containerID="ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b"
Jul 28 20:38:03.994930 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:38:03.994909    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b\": container with ID starting with ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b not found: ID does not exist" containerID="ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b"
Jul 28 20:38:03.994979 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:03.994938    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b"} err="failed to get container status \"ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b\": rpc error: code = NotFound desc = could not find container \"ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b\": container with ID starting with ebb16dae34204bce1ee6c23c3041f8dad7d48940ee3e2cfaef2497ea35ab4e0b not found: ID does not exist"
Jul 28 20:38:04.001185 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:04.001164    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"]
Jul 28 20:38:04.008527 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:04.008505    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ed417440-6844f67dcb-zr86b"]
Jul 28 20:38:04.023087 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:04.023066    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"]
Jul 28 20:38:04.029298 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:04.029277    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ed417440-55fc74d689-72lnm"]
Jul 28 20:38:04.411114 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:04.411084    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:04.652030 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:04.651996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:04.835846 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:04.835767    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ad8719c-b465-4f07-877e-892a00f20430" path="/var/lib/kubelet/pods/0ad8719c-b465-4f07-877e-892a00f20430/volumes"
Jul 28 20:38:04.836156 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:04.836142    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece2e62c-c433-4c98-8646-229cbec7abe5" path="/var/lib/kubelet/pods/ece2e62c-c433-4c98-8646-229cbec7abe5/volumes"
Jul 28 20:38:04.921919 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:04.921871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:05.232238 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:05.232194    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:05.857300 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:05.857269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:07.441419 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.441379    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:07.546079 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.546044    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:07.560219 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.560186    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"]
Jul 28 20:38:07.560565 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.560552    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad8719c-b465-4f07-877e-892a00f20430" containerName="payload-pre-processing"
Jul 28 20:38:07.560610 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.560566    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad8719c-b465-4f07-877e-892a00f20430" containerName="payload-pre-processing"
Jul 28 20:38:07.560610 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.560581    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece2e62c-c433-4c98-8646-229cbec7abe5" containerName="maas-api"
Jul 28 20:38:07.560610 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.560587    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece2e62c-c433-4c98-8646-229cbec7abe5" containerName="maas-api"
Jul 28 20:38:07.560707 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.560658    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece2e62c-c433-4c98-8646-229cbec7abe5" containerName="maas-api"
Jul 28 20:38:07.560707 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.560666    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad8719c-b465-4f07-877e-892a00f20430" containerName="payload-pre-processing"
Jul 28 20:38:07.564920 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.564900    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:07.567428 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.567410    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0fbea363-serving-cert\""
Jul 28 20:38:07.583346 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.583316    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"]
Jul 28 20:38:07.612433 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.612403    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14b84b7-7d45-4c75-8a20-75574bd4c537-openshift-service-ca\") pod \"maas-api-e2e-derive-0fbea363-89f57885c-fhw5d\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:07.612562 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.612448    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14b84b7-7d45-4c75-8a20-75574bd4c537-maas-api-tls\") pod \"maas-api-e2e-derive-0fbea363-89f57885c-fhw5d\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:07.612562 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.612487    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nfr7\" (UniqueName: \"kubernetes.io/projected/e14b84b7-7d45-4c75-8a20-75574bd4c537-kube-api-access-5nfr7\") pod \"maas-api-e2e-derive-0fbea363-89f57885c-fhw5d\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:07.642302 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.642273    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:07.713593 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.713517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14b84b7-7d45-4c75-8a20-75574bd4c537-openshift-service-ca\") pod \"maas-api-e2e-derive-0fbea363-89f57885c-fhw5d\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:07.713593 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.713561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14b84b7-7d45-4c75-8a20-75574bd4c537-maas-api-tls\") pod \"maas-api-e2e-derive-0fbea363-89f57885c-fhw5d\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:07.713761 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.713599    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfr7\" (UniqueName: \"kubernetes.io/projected/e14b84b7-7d45-4c75-8a20-75574bd4c537-kube-api-access-5nfr7\") pod \"maas-api-e2e-derive-0fbea363-89f57885c-fhw5d\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:07.713761 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:38:07.713721    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-serving-cert: secret "maas-api-e2e-derive-0fbea363-serving-cert" not found
Jul 28 20:38:07.713829 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:38:07.713789    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e14b84b7-7d45-4c75-8a20-75574bd4c537-maas-api-tls podName:e14b84b7-7d45-4c75-8a20-75574bd4c537 nodeName:}" failed. No retries permitted until 2026-07-28 20:38:08.213768322 +0000 UTC m=+1787.828005448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e14b84b7-7d45-4c75-8a20-75574bd4c537-maas-api-tls") pod "maas-api-e2e-derive-0fbea363-89f57885c-fhw5d" (UID: "e14b84b7-7d45-4c75-8a20-75574bd4c537") : secret "maas-api-e2e-derive-0fbea363-serving-cert" not found
Jul 28 20:38:07.714160 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.714140    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14b84b7-7d45-4c75-8a20-75574bd4c537-openshift-service-ca\") pod \"maas-api-e2e-derive-0fbea363-89f57885c-fhw5d\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:07.729010 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:07.728976    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfr7\" (UniqueName: \"kubernetes.io/projected/e14b84b7-7d45-4c75-8a20-75574bd4c537-kube-api-access-5nfr7\") pod \"maas-api-e2e-derive-0fbea363-89f57885c-fhw5d\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:08.061876 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.061789    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:08.218172 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.218130    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14b84b7-7d45-4c75-8a20-75574bd4c537-maas-api-tls\") pod \"maas-api-e2e-derive-0fbea363-89f57885c-fhw5d\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:08.220618 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.220596    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14b84b7-7d45-4c75-8a20-75574bd4c537-maas-api-tls\") pod \"maas-api-e2e-derive-0fbea363-89f57885c-fhw5d\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:08.478112 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.478078    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:08.522864 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.522832    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"]
Jul 28 20:38:08.526588 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.526562    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:08.532139 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.530043    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0fbea363-dockercfg-q7nlv\""
Jul 28 20:38:08.532139 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.531906    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0fbea363\""
Jul 28 20:38:08.549652 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.547704    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"]
Jul 28 20:38:08.621486 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.621454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1c22cd-507c-497d-969d-f3d7f99d2029-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw\" (UID: \"7b1c22cd-507c-497d-969d-f3d7f99d2029\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:08.621650 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.621494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2ttk\" (UniqueName: \"kubernetes.io/projected/7b1c22cd-507c-497d-969d-f3d7f99d2029-kube-api-access-j2ttk\") pod \"payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw\" (UID: \"7b1c22cd-507c-497d-969d-f3d7f99d2029\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:08.722549 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.722515    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1c22cd-507c-497d-969d-f3d7f99d2029-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw\" (UID: \"7b1c22cd-507c-497d-969d-f3d7f99d2029\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:08.722704 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.722555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ttk\" (UniqueName: \"kubernetes.io/projected/7b1c22cd-507c-497d-969d-f3d7f99d2029-kube-api-access-j2ttk\") pod \"payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw\" (UID: \"7b1c22cd-507c-497d-969d-f3d7f99d2029\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:08.723196 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.723177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1c22cd-507c-497d-969d-f3d7f99d2029-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw\" (UID: \"7b1c22cd-507c-497d-969d-f3d7f99d2029\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:08.731186 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.731121    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ttk\" (UniqueName: \"kubernetes.io/projected/7b1c22cd-507c-497d-969d-f3d7f99d2029-kube-api-access-j2ttk\") pod \"payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw\" (UID: \"7b1c22cd-507c-497d-969d-f3d7f99d2029\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:08.843444 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.843420    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"]
Jul 28 20:38:08.845001 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.844983    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:08.845771 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:38:08.845748    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14b84b7_7d45_4c75_8a20_75574bd4c537.slice/crio-0a480a7f3d147e633f0692a9d191c498874c930f90a70ec0e71374f86717b42b WatchSource:0}: Error finding container 0a480a7f3d147e633f0692a9d191c498874c930f90a70ec0e71374f86717b42b: Status 404 returned error can't find the container with id 0a480a7f3d147e633f0692a9d191c498874c930f90a70ec0e71374f86717b42b
Jul 28 20:38:08.990017 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:08.989949    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"]
Jul 28 20:38:08.993423 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:38:08.993397    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1c22cd_507c_497d_969d_f3d7f99d2029.slice/crio-b98fe67d12a15fccb3ba00d52e00eb44d7a84b4c906e2843e12879ba02c8f74f WatchSource:0}: Error finding container b98fe67d12a15fccb3ba00d52e00eb44d7a84b4c906e2843e12879ba02c8f74f: Status 404 returned error can't find the container with id b98fe67d12a15fccb3ba00d52e00eb44d7a84b4c906e2843e12879ba02c8f74f
Jul 28 20:38:09.006024 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:09.005988    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw" event={"ID":"7b1c22cd-507c-497d-969d-f3d7f99d2029","Type":"ContainerStarted","Data":"b98fe67d12a15fccb3ba00d52e00eb44d7a84b4c906e2843e12879ba02c8f74f"}
Jul 28 20:38:09.007009 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:09.006993    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d" event={"ID":"e14b84b7-7d45-4c75-8a20-75574bd4c537","Type":"ContainerStarted","Data":"0a480a7f3d147e633f0692a9d191c498874c930f90a70ec0e71374f86717b42b"}
Jul 28 20:38:09.748884 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:09.748853    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:09.809420 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:09.809390    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:10.013333 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:10.013213    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw" event={"ID":"7b1c22cd-507c-497d-969d-f3d7f99d2029","Type":"ContainerStarted","Data":"3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e"}
Jul 28 20:38:10.013333 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:10.013305    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:10.014604 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:10.014573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d" event={"ID":"e14b84b7-7d45-4c75-8a20-75574bd4c537","Type":"ContainerStarted","Data":"d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5"}
Jul 28 20:38:10.014747 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:10.014696    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:10.033751 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:10.033693    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw" podStartSLOduration=2.033675174 podStartE2EDuration="2.033675174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:38:10.030944115 +0000 UTC m=+1789.645181247" watchObservedRunningTime="2026-07-28 20:38:10.033675174 +0000 UTC m=+1789.647912308"
Jul 28 20:38:10.049470 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:10.049422    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d" podStartSLOduration=2.666984642 podStartE2EDuration="3.049405934s" totalImagesPullingTime="382.421292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:38:10.047666338 +0000 UTC m=+1789.661903472" watchObservedRunningTime="2026-07-28 20:38:10.049405934 +0000 UTC m=+1789.663643065"
Jul 28 20:38:16.024885 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:16.024806    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:21.021340 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:21.021312    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:38.262350 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.262302    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"]
Jul 28 20:38:38.263173 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.263145    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d" podUID="e14b84b7-7d45-4c75-8a20-75574bd4c537" containerName="maas-api" containerID="cri-o://d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5" gracePeriod=30
Jul 28 20:38:38.285374 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.285344    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:38.337671 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.337638    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"]
Jul 28 20:38:38.337969 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.337940    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw" podUID="7b1c22cd-507c-497d-969d-f3d7f99d2029" containerName="payload-pre-processing" containerID="cri-o://3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e" gracePeriod=30
Jul 28 20:38:38.382939 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.382898    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:38.483201 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.483169    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:38.583944 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.583919    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:38.587645 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.587625    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:38.617449 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.617426    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:38.674943 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.674914    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nfr7\" (UniqueName: \"kubernetes.io/projected/e14b84b7-7d45-4c75-8a20-75574bd4c537-kube-api-access-5nfr7\") pod \"e14b84b7-7d45-4c75-8a20-75574bd4c537\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") "
Jul 28 20:38:38.675110 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.674951    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14b84b7-7d45-4c75-8a20-75574bd4c537-openshift-service-ca\") pod \"e14b84b7-7d45-4c75-8a20-75574bd4c537\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") "
Jul 28 20:38:38.675110 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.675015    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14b84b7-7d45-4c75-8a20-75574bd4c537-maas-api-tls\") pod \"e14b84b7-7d45-4c75-8a20-75574bd4c537\" (UID: \"e14b84b7-7d45-4c75-8a20-75574bd4c537\") "
Jul 28 20:38:38.675407 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.675379    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14b84b7-7d45-4c75-8a20-75574bd4c537-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e14b84b7-7d45-4c75-8a20-75574bd4c537" (UID: "e14b84b7-7d45-4c75-8a20-75574bd4c537"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:38:38.676921 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.676896    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14b84b7-7d45-4c75-8a20-75574bd4c537-kube-api-access-5nfr7" (OuterVolumeSpecName: "kube-api-access-5nfr7") pod "e14b84b7-7d45-4c75-8a20-75574bd4c537" (UID: "e14b84b7-7d45-4c75-8a20-75574bd4c537"). InnerVolumeSpecName "kube-api-access-5nfr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:38:38.677030 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.676927    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14b84b7-7d45-4c75-8a20-75574bd4c537-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e14b84b7-7d45-4c75-8a20-75574bd4c537" (UID: "e14b84b7-7d45-4c75-8a20-75574bd4c537"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:38:38.695822 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.695800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:38.776028 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.776006    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2ttk\" (UniqueName: \"kubernetes.io/projected/7b1c22cd-507c-497d-969d-f3d7f99d2029-kube-api-access-j2ttk\") pod \"7b1c22cd-507c-497d-969d-f3d7f99d2029\" (UID: \"7b1c22cd-507c-497d-969d-f3d7f99d2029\") "
Jul 28 20:38:38.776149 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.776038    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1c22cd-507c-497d-969d-f3d7f99d2029-plugins-config-volume\") pod \"7b1c22cd-507c-497d-969d-f3d7f99d2029\" (UID: \"7b1c22cd-507c-497d-969d-f3d7f99d2029\") "
Jul 28 20:38:38.776258 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.776220    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14b84b7-7d45-4c75-8a20-75574bd4c537-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:38:38.776306 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.776264    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14b84b7-7d45-4c75-8a20-75574bd4c537-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:38:38.776306 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.776278    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nfr7\" (UniqueName: \"kubernetes.io/projected/e14b84b7-7d45-4c75-8a20-75574bd4c537-kube-api-access-5nfr7\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:38:38.776447 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.776426    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1c22cd-507c-497d-969d-f3d7f99d2029-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b1c22cd-507c-497d-969d-f3d7f99d2029" (UID: "7b1c22cd-507c-497d-969d-f3d7f99d2029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:38:38.777899 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.777878    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1c22cd-507c-497d-969d-f3d7f99d2029-kube-api-access-j2ttk" (OuterVolumeSpecName: "kube-api-access-j2ttk") pod "7b1c22cd-507c-497d-969d-f3d7f99d2029" (UID: "7b1c22cd-507c-497d-969d-f3d7f99d2029"). InnerVolumeSpecName "kube-api-access-j2ttk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:38:38.877669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.877644    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2ttk\" (UniqueName: \"kubernetes.io/projected/7b1c22cd-507c-497d-969d-f3d7f99d2029-kube-api-access-j2ttk\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:38:38.877669 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.877670    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1c22cd-507c-497d-969d-f3d7f99d2029-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:38:38.902644 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:38.902614    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:39.163778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.163746    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b1c22cd-507c-497d-969d-f3d7f99d2029" containerID="3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e" exitCode=0
Jul 28 20:38:39.163941 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.163798    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"
Jul 28 20:38:39.163941 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.163832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw" event={"ID":"7b1c22cd-507c-497d-969d-f3d7f99d2029","Type":"ContainerDied","Data":"3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e"}
Jul 28 20:38:39.163941 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.163893    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw" event={"ID":"7b1c22cd-507c-497d-969d-f3d7f99d2029","Type":"ContainerDied","Data":"b98fe67d12a15fccb3ba00d52e00eb44d7a84b4c906e2843e12879ba02c8f74f"}
Jul 28 20:38:39.163941 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.163921    2575 scope.go:117] "RemoveContainer" containerID="3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e"
Jul 28 20:38:39.165333 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.165313    2575 generic.go:356] "Generic (PLEG): container finished" podID="e14b84b7-7d45-4c75-8a20-75574bd4c537" containerID="d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5" exitCode=0
Jul 28 20:38:39.165456 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.165345    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d" event={"ID":"e14b84b7-7d45-4c75-8a20-75574bd4c537","Type":"ContainerDied","Data":"d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5"}
Jul 28 20:38:39.165456 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.165364    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"
Jul 28 20:38:39.165456 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.165388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d" event={"ID":"e14b84b7-7d45-4c75-8a20-75574bd4c537","Type":"ContainerDied","Data":"0a480a7f3d147e633f0692a9d191c498874c930f90a70ec0e71374f86717b42b"}
Jul 28 20:38:39.174382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.174366    2575 scope.go:117] "RemoveContainer" containerID="3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e"
Jul 28 20:38:39.174660 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:38:39.174637    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e\": container with ID starting with 3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e not found: ID does not exist" containerID="3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e"
Jul 28 20:38:39.174756 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.174667    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e"} err="failed to get container status \"3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e\": rpc error: code = NotFound desc = could not find container \"3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e\": container with ID starting with 3dcb5ae863031df1a0217dacd8516b5f41f07bae90977916cb9f915ba575943e not found: ID does not exist"
Jul 28 20:38:39.174756 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.174685    2575 scope.go:117] "RemoveContainer" containerID="d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5"
Jul 28 20:38:39.182271 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.182253    2575 scope.go:117] "RemoveContainer" containerID="d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5"
Jul 28 20:38:39.182531 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:38:39.182504    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5\": container with ID starting with d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5 not found: ID does not exist" containerID="d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5"
Jul 28 20:38:39.182582 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.182539    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5"} err="failed to get container status \"d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5\": rpc error: code = NotFound desc = could not find container \"d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5\": container with ID starting with d22d7757ca29f3a4c3bb573e425002edf2797d83c497e81bd735d7a15c0703e5 not found: ID does not exist"
Jul 28 20:38:39.186683 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.186660    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"]
Jul 28 20:38:39.193793 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.193775    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0fbea363-89f57885c-fhw5d"]
Jul 28 20:38:39.205172 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.205153    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"]
Jul 28 20:38:39.209198 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.209179    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0fbea363-9d9d4665b-rj8cw"]
Jul 28 20:38:39.420615 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.420536    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:39.512871 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.512823    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:39.615192 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:39.615158    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:40.837619 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:40.837584    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1c22cd-507c-497d-969d-f3d7f99d2029" path="/var/lib/kubelet/pods/7b1c22cd-507c-497d-969d-f3d7f99d2029/volumes"
Jul 28 20:38:40.837992 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:40.837980    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e14b84b7-7d45-4c75-8a20-75574bd4c537" path="/var/lib/kubelet/pods/e14b84b7-7d45-4c75-8a20-75574bd4c537/volumes"
Jul 28 20:38:41.070933 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:41.070902    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:47.780310 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:47.780270    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:47.885430 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:47.885399    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:47.989987 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:47.989953    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:48.117263 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:48.117157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:38:48.580164 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:38:48.580135    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:14.552269 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:14.552183    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:21.213131 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:21.213075    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:21.318641 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:21.318562    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:21.458387 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:21.458352    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:21.587005 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:21.586917    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:37.350014 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:37.349980    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:43.168957 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:43.168855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:43.273916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:43.273886    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:43.378824 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:43.378780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:53.932207 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:53.932161    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:39:54.743627 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:39:54.743594    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:03.056023 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:03.055987    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:03.311242 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:03.311152    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:14.668818 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:14.668779    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:14.771129 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:14.771094    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:14.871939 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:14.871907    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:17.117183 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:17.117148    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:23.923137 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:23.923099    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:24.030940 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:24.030903    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:24.685895 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:24.685863    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:30.609363 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:30.609329    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:30.708043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:30.707998    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:30.860096 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:30.860016    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:31.234642 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:31.234610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:31.391117 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:31.391085    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:34.042626 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.042597    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:34.143283 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.143220    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"]
Jul 28 20:40:34.143788 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.143772    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14b84b7-7d45-4c75-8a20-75574bd4c537" containerName="maas-api"
Jul 28 20:40:34.143788 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.143790    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14b84b7-7d45-4c75-8a20-75574bd4c537" containerName="maas-api"
Jul 28 20:40:34.143950 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.143804    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1c22cd-507c-497d-969d-f3d7f99d2029" containerName="payload-pre-processing"
Jul 28 20:40:34.143950 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.143812    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1c22cd-507c-497d-969d-f3d7f99d2029" containerName="payload-pre-processing"
Jul 28 20:40:34.143950 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.143888    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1c22cd-507c-497d-969d-f3d7f99d2029" containerName="payload-pre-processing"
Jul 28 20:40:34.143950 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.143898    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14b84b7-7d45-4c75-8a20-75574bd4c537" containerName="maas-api"
Jul 28 20:40:34.147265 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.147244    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:34.149591 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.149565    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fd997c17-serving-cert\""
Jul 28 20:40:34.159490 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.159463    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"]
Jul 28 20:40:34.211136 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.211100    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7953a114-1463-4a0d-bb78-ac12c3304699-openshift-service-ca\") pod \"maas-api-e2e-mt-fd997c17-84d87448fc-8h26s\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:34.211315 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.211141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvpsv\" (UniqueName: \"kubernetes.io/projected/7953a114-1463-4a0d-bb78-ac12c3304699-kube-api-access-hvpsv\") pod \"maas-api-e2e-mt-fd997c17-84d87448fc-8h26s\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:34.211315 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.211267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7953a114-1463-4a0d-bb78-ac12c3304699-maas-api-tls\") pod \"maas-api-e2e-mt-fd997c17-84d87448fc-8h26s\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:34.312476 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.312388    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7953a114-1463-4a0d-bb78-ac12c3304699-openshift-service-ca\") pod \"maas-api-e2e-mt-fd997c17-84d87448fc-8h26s\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:34.312476 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.312433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvpsv\" (UniqueName: \"kubernetes.io/projected/7953a114-1463-4a0d-bb78-ac12c3304699-kube-api-access-hvpsv\") pod \"maas-api-e2e-mt-fd997c17-84d87448fc-8h26s\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:34.312699 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.312504    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7953a114-1463-4a0d-bb78-ac12c3304699-maas-api-tls\") pod \"maas-api-e2e-mt-fd997c17-84d87448fc-8h26s\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:34.312699 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:40:34.312657    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-serving-cert: secret "maas-api-e2e-mt-fd997c17-serving-cert" not found
Jul 28 20:40:34.312814 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:40:34.312727    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7953a114-1463-4a0d-bb78-ac12c3304699-maas-api-tls podName:7953a114-1463-4a0d-bb78-ac12c3304699 nodeName:}" failed. No retries permitted until 2026-07-28 20:40:34.812706519 +0000 UTC m=+1934.426943632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7953a114-1463-4a0d-bb78-ac12c3304699-maas-api-tls") pod "maas-api-e2e-mt-fd997c17-84d87448fc-8h26s" (UID: "7953a114-1463-4a0d-bb78-ac12c3304699") : secret "maas-api-e2e-mt-fd997c17-serving-cert" not found
Jul 28 20:40:34.313116 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.313095    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7953a114-1463-4a0d-bb78-ac12c3304699-openshift-service-ca\") pod \"maas-api-e2e-mt-fd997c17-84d87448fc-8h26s\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:34.329818 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.329793    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvpsv\" (UniqueName: \"kubernetes.io/projected/7953a114-1463-4a0d-bb78-ac12c3304699-kube-api-access-hvpsv\") pod \"maas-api-e2e-mt-fd997c17-84d87448fc-8h26s\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:34.815705 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.815670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7953a114-1463-4a0d-bb78-ac12c3304699-maas-api-tls\") pod \"maas-api-e2e-mt-fd997c17-84d87448fc-8h26s\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:34.818042 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:34.818014    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7953a114-1463-4a0d-bb78-ac12c3304699-maas-api-tls\") pod \"maas-api-e2e-mt-fd997c17-84d87448fc-8h26s\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:35.058561 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:35.058519    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:35.205287 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:35.205252    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"]
Jul 28 20:40:35.207407 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:40:35.207381    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7953a114_1463_4a0d_bb78_ac12c3304699.slice/crio-be8dd829b24c98fe2d05a3ee7f80fc7673c8776961ab4fa44c0995ddd6b79b29 WatchSource:0}: Error finding container be8dd829b24c98fe2d05a3ee7f80fc7673c8776961ab4fa44c0995ddd6b79b29: Status 404 returned error can't find the container with id be8dd829b24c98fe2d05a3ee7f80fc7673c8776961ab4fa44c0995ddd6b79b29
Jul 28 20:40:35.516748 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:35.516718    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:35.710314 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:35.710282    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s" event={"ID":"7953a114-1463-4a0d-bb78-ac12c3304699","Type":"ContainerStarted","Data":"be8dd829b24c98fe2d05a3ee7f80fc7673c8776961ab4fa44c0995ddd6b79b29"}
Jul 28 20:40:36.717313 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:36.717279    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s" event={"ID":"7953a114-1463-4a0d-bb78-ac12c3304699","Type":"ContainerStarted","Data":"a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53"}
Jul 28 20:40:36.717828 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:36.717386    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:36.741337 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:36.741275    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s" podStartSLOduration=2.215918037 podStartE2EDuration="2.741254412s" totalImagesPullingTime="525.336375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:40:36.739599925 +0000 UTC m=+1936.353837060" watchObservedRunningTime="2026-07-28 20:40:36.741254412 +0000 UTC m=+1936.355491546"
Jul 28 20:40:42.727874 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:42.727838    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:40:58.948307 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:58.948270    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:59.657199 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:59.657161    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:40:59.741904 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:40:59.741871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:01.366550 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:01.366514    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:01.468778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:01.468742    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:05.831686 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:05.831656    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:05.931901 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:05.931871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:16.881420 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:16.881212    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"]
Jul 28 20:41:16.882057 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:16.882015    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s" podUID="7953a114-1463-4a0d-bb78-ac12c3304699" containerName="maas-api" containerID="cri-o://a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53" gracePeriod=30
Jul 28 20:41:16.920481 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:16.920455    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:17.004197 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.004169    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:17.094918 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.094889    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:17.138098 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.138034    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:41:17.188503 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.188476    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7953a114-1463-4a0d-bb78-ac12c3304699-maas-api-tls\") pod \"7953a114-1463-4a0d-bb78-ac12c3304699\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") "
Jul 28 20:41:17.188681 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.188564    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7953a114-1463-4a0d-bb78-ac12c3304699-openshift-service-ca\") pod \"7953a114-1463-4a0d-bb78-ac12c3304699\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") "
Jul 28 20:41:17.188681 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.188607    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvpsv\" (UniqueName: \"kubernetes.io/projected/7953a114-1463-4a0d-bb78-ac12c3304699-kube-api-access-hvpsv\") pod \"7953a114-1463-4a0d-bb78-ac12c3304699\" (UID: \"7953a114-1463-4a0d-bb78-ac12c3304699\") "
Jul 28 20:41:17.189035 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.188992    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7953a114-1463-4a0d-bb78-ac12c3304699-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7953a114-1463-4a0d-bb78-ac12c3304699" (UID: "7953a114-1463-4a0d-bb78-ac12c3304699"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:41:17.190724 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.190700    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7953a114-1463-4a0d-bb78-ac12c3304699-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7953a114-1463-4a0d-bb78-ac12c3304699" (UID: "7953a114-1463-4a0d-bb78-ac12c3304699"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:41:17.190724 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.190716    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7953a114-1463-4a0d-bb78-ac12c3304699-kube-api-access-hvpsv" (OuterVolumeSpecName: "kube-api-access-hvpsv") pod "7953a114-1463-4a0d-bb78-ac12c3304699" (UID: "7953a114-1463-4a0d-bb78-ac12c3304699"). InnerVolumeSpecName "kube-api-access-hvpsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:41:17.204110 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.204084    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:17.289324 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.289291    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7953a114-1463-4a0d-bb78-ac12c3304699-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:41:17.289324 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.289325    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvpsv\" (UniqueName: \"kubernetes.io/projected/7953a114-1463-4a0d-bb78-ac12c3304699-kube-api-access-hvpsv\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:41:17.289509 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.289342    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7953a114-1463-4a0d-bb78-ac12c3304699-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:41:17.565323 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.565290    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:17.904372 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.904338    2575 generic.go:356] "Generic (PLEG): container finished" podID="7953a114-1463-4a0d-bb78-ac12c3304699" containerID="a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53" exitCode=0
Jul 28 20:41:17.904743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.904399    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"
Jul 28 20:41:17.904743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.904387    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s" event={"ID":"7953a114-1463-4a0d-bb78-ac12c3304699","Type":"ContainerDied","Data":"a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53"}
Jul 28 20:41:17.904743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.904518    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s" event={"ID":"7953a114-1463-4a0d-bb78-ac12c3304699","Type":"ContainerDied","Data":"be8dd829b24c98fe2d05a3ee7f80fc7673c8776961ab4fa44c0995ddd6b79b29"}
Jul 28 20:41:17.904743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.904548    2575 scope.go:117] "RemoveContainer" containerID="a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53"
Jul 28 20:41:17.913658 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.913638    2575 scope.go:117] "RemoveContainer" containerID="a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53"
Jul 28 20:41:17.913913 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:41:17.913895    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53\": container with ID starting with a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53 not found: ID does not exist" containerID="a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53"
Jul 28 20:41:17.913968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.913919    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53"} err="failed to get container status \"a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53\": rpc error: code = NotFound desc = could not find container \"a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53\": container with ID starting with a22ed4de928083f4e888e1dcdb8c9cb5c9a5a18c7641126842d32fc8037b7b53 not found: ID does not exist"
Jul 28 20:41:17.925385 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.925362    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"]
Jul 28 20:41:17.929075 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:17.929055    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd997c17-84d87448fc-8h26s"]
Jul 28 20:41:18.837410 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:18.837377    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7953a114-1463-4a0d-bb78-ac12c3304699" path="/var/lib/kubelet/pods/7953a114-1463-4a0d-bb78-ac12c3304699/volumes"
Jul 28 20:41:20.062966 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:20.062923    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:23.650102 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:23.650064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:23.761156 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:23.761118    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:23.953150 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:23.953075    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:24.185056 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:24.185025    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:25.517445 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:25.517415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:25.589335 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:25.589303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:30.797289 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:30.797256    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:30.904355 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:30.904312    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:30.996749 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:30.996718    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:31.164676 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:31.164644    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:37.457173 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:37.457138    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:37.557767 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:37.557731    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:44.292715 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:44.292677    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:44.615148 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:44.615062    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:44.703589 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:44.703557    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:47.250135 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.250092    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:47.444098 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.444055    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:47.481215 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.481183    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"]
Jul 28 20:41:47.481764 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.481749    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7953a114-1463-4a0d-bb78-ac12c3304699" containerName="maas-api"
Jul 28 20:41:47.481807 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.481767    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7953a114-1463-4a0d-bb78-ac12c3304699" containerName="maas-api"
Jul 28 20:41:47.481869 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.481858    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7953a114-1463-4a0d-bb78-ac12c3304699" containerName="maas-api"
Jul 28 20:41:47.485287 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.485269    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.488973 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.488953    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0cb82b97-serving-cert\""
Jul 28 20:41:47.512334 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.512264    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"]
Jul 28 20:41:47.609487 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.609458    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:47.648008 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.647972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/791d3033-05b5-4c4d-ba85-70f5d21a5d53-openshift-service-ca\") pod \"maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.648183 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.648014    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/791d3033-05b5-4c4d-ba85-70f5d21a5d53-maas-api-tls\") pod \"maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.648183 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.648047    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsdbv\" (UniqueName: \"kubernetes.io/projected/791d3033-05b5-4c4d-ba85-70f5d21a5d53-kube-api-access-dsdbv\") pod \"maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.712845 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.712790    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:47.748964 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.748933    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/791d3033-05b5-4c4d-ba85-70f5d21a5d53-openshift-service-ca\") pod \"maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.749152 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.748972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/791d3033-05b5-4c4d-ba85-70f5d21a5d53-maas-api-tls\") pod \"maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.749152 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.749001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsdbv\" (UniqueName: \"kubernetes.io/projected/791d3033-05b5-4c4d-ba85-70f5d21a5d53-kube-api-access-dsdbv\") pod \"maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.749728 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.749701    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/791d3033-05b5-4c4d-ba85-70f5d21a5d53-openshift-service-ca\") pod \"maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.751642 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.751614    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/791d3033-05b5-4c4d-ba85-70f5d21a5d53-maas-api-tls\") pod \"maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.757947 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.757922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsdbv\" (UniqueName: \"kubernetes.io/projected/791d3033-05b5-4c4d-ba85-70f5d21a5d53-kube-api-access-dsdbv\") pod \"maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.795082 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.794985    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:47.926508 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.926480    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"]
Jul 28 20:41:47.929184 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:41:47.929153    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791d3033_05b5_4c4d_ba85_70f5d21a5d53.slice/crio-5bf307b2d80f3ee898463569886f7f753b9a6034b7c4af913079782458be33f7 WatchSource:0}: Error finding container 5bf307b2d80f3ee898463569886f7f753b9a6034b7c4af913079782458be33f7: Status 404 returned error can't find the container with id 5bf307b2d80f3ee898463569886f7f753b9a6034b7c4af913079782458be33f7
Jul 28 20:41:47.930781 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:47.930759    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:41:48.043623 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:48.043584    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69" event={"ID":"791d3033-05b5-4c4d-ba85-70f5d21a5d53","Type":"ContainerStarted","Data":"5bf307b2d80f3ee898463569886f7f753b9a6034b7c4af913079782458be33f7"}
Jul 28 20:41:48.204732 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:48.204699    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:48.349796 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:48.349766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:41:49.049313 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:49.049267    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69" event={"ID":"791d3033-05b5-4c4d-ba85-70f5d21a5d53","Type":"ContainerStarted","Data":"702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2"}
Jul 28 20:41:49.049494 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:49.049383    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:41:49.067141 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:49.067094    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69" podStartSLOduration=1.6055058739999999 podStartE2EDuration="2.067080212s" totalImagesPullingTime="461.574338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:41:49.06657846 +0000 UTC m=+2008.680815618" watchObservedRunningTime="2026-07-28 20:41:49.067080212 +0000 UTC m=+2008.681317344"
Jul 28 20:41:55.060477 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:41:55.060447    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:42:12.209965 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:12.209928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:13.142178 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:13.142131    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:23.503463 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.503427    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"]
Jul 28 20:42:23.507021 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.507004    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.509889 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.509862    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0cb82b97\"/\"openshift-service-ca.crt\""
Jul 28 20:42:23.510502 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.510481    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0cb82b97\"/\"kube-root-ca.crt\""
Jul 28 20:42:23.510568 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.510512    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0cb82b97\"/\"e2e-lifecycle-model-0cb82b97-kserve-self-signed-certs\""
Jul 28 20:42:23.510664 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.510642    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0cb82b97\"/\"default-dockercfg-bshdm\""
Jul 28 20:42:23.544308 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.544280    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"]
Jul 28 20:42:23.552403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.552377    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tls-certs\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.552515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.552447    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-home\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.552515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.552471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-model-cache\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.552515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.552488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvgf\" (UniqueName: \"kubernetes.io/projected/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-kube-api-access-vqvgf\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.552667 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.552513    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-dshm\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.552667 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.552537    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tmp-dir\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.597269 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.597239    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:42:23.653645 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.653610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tls-certs\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.653794 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.653664    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-home\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.653794 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.653776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-model-cache\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.653883 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.653813    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvgf\" (UniqueName: \"kubernetes.io/projected/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-kube-api-access-vqvgf\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.653883 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.653843    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-dshm\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.653883 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.653871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tmp-dir\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.654333 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.654304    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-model-cache\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.654333 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.654323    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-home\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.654478 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.654389    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tmp-dir\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.656135 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.656110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-dshm\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.656302 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.656286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tls-certs\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.666005 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.665976    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvgf\" (UniqueName: \"kubernetes.io/projected/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-kube-api-access-vqvgf\") pod \"e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") " pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.816982 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.816899    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:23.960054 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:23.960026    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"]
Jul 28 20:42:23.962557 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:42:23.962526    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5fbdce2_31e1_4dc6_8ee9_582b0c41c973.slice/crio-41931e4522e3e385866fc2545d72d23fbd3b53f8e795d5366408c9f244de88e7 WatchSource:0}: Error finding container 41931e4522e3e385866fc2545d72d23fbd3b53f8e795d5366408c9f244de88e7: Status 404 returned error can't find the container with id 41931e4522e3e385866fc2545d72d23fbd3b53f8e795d5366408c9f244de88e7
Jul 28 20:42:24.215345 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:24.215314    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt" event={"ID":"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973","Type":"ContainerStarted","Data":"2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc"}
Jul 28 20:42:24.215508 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:24.215355    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt" event={"ID":"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973","Type":"ContainerStarted","Data":"41931e4522e3e385866fc2545d72d23fbd3b53f8e795d5366408c9f244de88e7"}
Jul 28 20:42:24.242434 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:24.242388    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt" podStartSLOduration=1.242374001 podStartE2EDuration="1.242374001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:42:24.240156482 +0000 UTC m=+2043.854393617" watchObservedRunningTime="2026-07-28 20:42:24.242374001 +0000 UTC m=+2043.856611137"
Jul 28 20:42:33.817646 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:33.817609    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:33.818099 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:33.817659    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:33.830213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:33.830189    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:42:34.273311 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:42:34.273280    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:43:00.821995 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:00.821912    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:00.923070 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:00.923021    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:02.293780 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.293747    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"]
Jul 28 20:43:02.294154 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.293984    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69" podUID="791d3033-05b5-4c4d-ba85-70f5d21a5d53" containerName="maas-api" containerID="cri-o://702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2" gracePeriod=30
Jul 28 20:43:02.318096 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.318024    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:02.398590 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.398547    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:02.486983 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.486953    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:02.547933 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.547873    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:02.549567 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.549550    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:43:02.574583 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.574560    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsdbv\" (UniqueName: \"kubernetes.io/projected/791d3033-05b5-4c4d-ba85-70f5d21a5d53-kube-api-access-dsdbv\") pod \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") "
Jul 28 20:43:02.574713 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.574619    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/791d3033-05b5-4c4d-ba85-70f5d21a5d53-maas-api-tls\") pod \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") "
Jul 28 20:43:02.574780 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.574712    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/791d3033-05b5-4c4d-ba85-70f5d21a5d53-openshift-service-ca\") pod \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\" (UID: \"791d3033-05b5-4c4d-ba85-70f5d21a5d53\") "
Jul 28 20:43:02.575106 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.575072    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791d3033-05b5-4c4d-ba85-70f5d21a5d53-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "791d3033-05b5-4c4d-ba85-70f5d21a5d53" (UID: "791d3033-05b5-4c4d-ba85-70f5d21a5d53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:43:02.576633 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.576610    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791d3033-05b5-4c4d-ba85-70f5d21a5d53-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "791d3033-05b5-4c4d-ba85-70f5d21a5d53" (UID: "791d3033-05b5-4c4d-ba85-70f5d21a5d53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:43:02.576741 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.576665    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791d3033-05b5-4c4d-ba85-70f5d21a5d53-kube-api-access-dsdbv" (OuterVolumeSpecName: "kube-api-access-dsdbv") pod "791d3033-05b5-4c4d-ba85-70f5d21a5d53" (UID: "791d3033-05b5-4c4d-ba85-70f5d21a5d53"). InnerVolumeSpecName "kube-api-access-dsdbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:43:02.675744 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.675707    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsdbv\" (UniqueName: \"kubernetes.io/projected/791d3033-05b5-4c4d-ba85-70f5d21a5d53-kube-api-access-dsdbv\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:43:02.675744 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.675737    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/791d3033-05b5-4c4d-ba85-70f5d21a5d53-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:43:02.675744 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:02.675746    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/791d3033-05b5-4c4d-ba85-70f5d21a5d53-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:43:03.406674 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:03.406638    2575 generic.go:356] "Generic (PLEG): container finished" podID="791d3033-05b5-4c4d-ba85-70f5d21a5d53" containerID="702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2" exitCode=0
Jul 28 20:43:03.407135 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:03.406677    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"
Jul 28 20:43:03.407135 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:03.406701    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69" event={"ID":"791d3033-05b5-4c4d-ba85-70f5d21a5d53","Type":"ContainerDied","Data":"702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2"}
Jul 28 20:43:03.407135 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:03.406742    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69" event={"ID":"791d3033-05b5-4c4d-ba85-70f5d21a5d53","Type":"ContainerDied","Data":"5bf307b2d80f3ee898463569886f7f753b9a6034b7c4af913079782458be33f7"}
Jul 28 20:43:03.407135 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:03.406763    2575 scope.go:117] "RemoveContainer" containerID="702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2"
Jul 28 20:43:03.415153 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:03.415138    2575 scope.go:117] "RemoveContainer" containerID="702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2"
Jul 28 20:43:03.415414 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:43:03.415396    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2\": container with ID starting with 702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2 not found: ID does not exist" containerID="702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2"
Jul 28 20:43:03.415472 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:03.415423    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2"} err="failed to get container status \"702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2\": rpc error: code = NotFound desc = could not find container \"702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2\": container with ID starting with 702f941dfc9cda48ea398bc17eb0568f93fcba80494dc2b96c5cd209e76b63b2 not found: ID does not exist"
Jul 28 20:43:03.424630 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:03.424607    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"]
Jul 28 20:43:03.432116 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:03.432093    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0cb82b97-7d9f9df476-zkh69"]
Jul 28 20:43:04.835756 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:04.835726    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791d3033-05b5-4c4d-ba85-70f5d21a5d53" path="/var/lib/kubelet/pods/791d3033-05b5-4c4d-ba85-70f5d21a5d53/volumes"
Jul 28 20:43:05.814887 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:05.814854    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:10.793156 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:10.793120    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"]
Jul 28 20:43:10.884973 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:10.884941    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:25.812493 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:25.812429    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt" podUID="a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" containerName="main" containerID="cri-o://2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc" gracePeriod=45
Jul 28 20:43:26.072061 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.072005    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:43:26.183967 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.183937    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-model-cache\") pod \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") "
Jul 28 20:43:26.184151 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.183973    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-home\") pod \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") "
Jul 28 20:43:26.184151 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.183995    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tls-certs\") pod \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") "
Jul 28 20:43:26.184151 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.184009    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-dshm\") pod \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") "
Jul 28 20:43:26.184151 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.184050    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tmp-dir\") pod \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") "
Jul 28 20:43:26.184151 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.184120    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqvgf\" (UniqueName: \"kubernetes.io/projected/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-kube-api-access-vqvgf\") pod \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\" (UID: \"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973\") "
Jul 28 20:43:26.184460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.184255    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" (UID: "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:43:26.184460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.184272    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-home" (OuterVolumeSpecName: "home") pod "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" (UID: "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:43:26.184460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.184352    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" (UID: "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:43:26.184460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.184434    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-model-cache\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:43:26.184460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.184450    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-home\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:43:26.184460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.184460    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tmp-dir\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:43:26.186321 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.186288    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-dshm" (OuterVolumeSpecName: "dshm") pod "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" (UID: "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:43:26.186321 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.186301    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" (UID: "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:43:26.186462 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.186352    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-kube-api-access-vqvgf" (OuterVolumeSpecName: "kube-api-access-vqvgf") pod "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" (UID: "a5fbdce2-31e1-4dc6-8ee9-582b0c41c973"). InnerVolumeSpecName "kube-api-access-vqvgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:43:26.285776 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.285741    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-tls-certs\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:43:26.285776 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.285770    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-dshm\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:43:26.285776 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.285782    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvgf\" (UniqueName: \"kubernetes.io/projected/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973-kube-api-access-vqvgf\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:43:26.513032 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.512999    2575 generic.go:356] "Generic (PLEG): container finished" podID="a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" containerID="2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc" exitCode=0
Jul 28 20:43:26.513267 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.513041    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"
Jul 28 20:43:26.513267 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.513088    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt" event={"ID":"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973","Type":"ContainerDied","Data":"2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc"}
Jul 28 20:43:26.513267 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.513138    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt" event={"ID":"a5fbdce2-31e1-4dc6-8ee9-582b0c41c973","Type":"ContainerDied","Data":"41931e4522e3e385866fc2545d72d23fbd3b53f8e795d5366408c9f244de88e7"}
Jul 28 20:43:26.513267 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.513163    2575 scope.go:117] "RemoveContainer" containerID="2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc"
Jul 28 20:43:26.522428 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.522413    2575 scope.go:117] "RemoveContainer" containerID="2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc"
Jul 28 20:43:26.522670 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:43:26.522652    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc\": container with ID starting with 2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc not found: ID does not exist" containerID="2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc"
Jul 28 20:43:26.522717 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.522680    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc"} err="failed to get container status \"2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc\": rpc error: code = NotFound desc = could not find container \"2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc\": container with ID starting with 2c2c708e237d94dfe828f4247d30a3c19bcfaa37b84cce24d7882bf58c58c0cc not found: ID does not exist"
Jul 28 20:43:26.534909 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.534888    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"]
Jul 28 20:43:26.539570 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.539549    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0cb82b97/e2e-lifecycle-model-0cb82b97-kserve-5cfdc594cf-7n5bt"]
Jul 28 20:43:26.835475 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:26.835389    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" path="/var/lib/kubelet/pods/a5fbdce2-31e1-4dc6-8ee9-582b0c41c973/volumes"
Jul 28 20:43:47.918505 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:47.918470    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:48.661157 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:48.661127    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:54.450949 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:54.450901    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:54.556157 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:54.556116    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:54.760245 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:54.760147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:55.042190 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:55.042095    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:56.315880 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:56.315844    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:56.484960 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:56.484922    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:57.401525 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:57.401487    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:57.596385 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:57.596355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:43:59.144537 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:43:59.144500    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:05.383434 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:05.383395    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:05.466382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:05.466351    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:06.091892 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:06.091859    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:13.098782 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:13.098706    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:13.206180 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:13.206143    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:13.301753 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:13.301696    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:23.882062 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:23.882022    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:24.229028 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:24.228991    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:45.209656 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:45.209623    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:52.702991 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:52.702953    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:52.804072 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:52.804043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:53.079119 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:53.079027    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:53.440829 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:53.440778    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:53.510802 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:53.510767    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:56.387433 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.387398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:56.478740 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.478703    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:56.506501 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.506474    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"]
Jul 28 20:44:56.506838 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.506827    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791d3033-05b5-4c4d-ba85-70f5d21a5d53" containerName="maas-api"
Jul 28 20:44:56.506882 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.506840    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="791d3033-05b5-4c4d-ba85-70f5d21a5d53" containerName="maas-api"
Jul 28 20:44:56.506882 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.506850    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" containerName="main"
Jul 28 20:44:56.506882 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.506856    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" containerName="main"
Jul 28 20:44:56.506979 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.506927    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="791d3033-05b5-4c4d-ba85-70f5d21a5d53" containerName="maas-api"
Jul 28 20:44:56.506979 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.506936    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5fbdce2-31e1-4dc6-8ee9-582b0c41c973" containerName="main"
Jul 28 20:44:56.509863 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.509843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:56.512389 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.512372    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d82230-serving-cert\""
Jul 28 20:44:56.521696 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.521670    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"]
Jul 28 20:44:56.546096 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.546060    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:56.546294 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.546112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-maas-api-tls\") pod \"maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:56.546294 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.546136    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpxql\" (UniqueName: \"kubernetes.io/projected/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-kube-api-access-mpxql\") pod \"maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:56.581682 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.581643    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:56.647185 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.647163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:56.647392 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.647200    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-maas-api-tls\") pod \"maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:56.647392 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.647219    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpxql\" (UniqueName: \"kubernetes.io/projected/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-kube-api-access-mpxql\") pod \"maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:56.647392 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:44:56.647346    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-serving-cert: secret "maas-api-e2e-inf-a-d82230-serving-cert" not found
Jul 28 20:44:56.647571 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:44:56.647436    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-maas-api-tls podName:c475ec2e-3a7a-4d54-80f0-71a4379b1fdd nodeName:}" failed. No retries permitted until 2026-07-28 20:44:57.147411682 +0000 UTC m=+2196.761648814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-maas-api-tls") pod "maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs" (UID: "c475ec2e-3a7a-4d54-80f0-71a4379b1fdd") : secret "maas-api-e2e-inf-a-d82230-serving-cert" not found
Jul 28 20:44:56.647827 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.647805    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:56.662458 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.662435    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpxql\" (UniqueName: \"kubernetes.io/projected/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-kube-api-access-mpxql\") pod \"maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:56.845384 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:56.845348    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:57.092959 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.092868    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:57.150794 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.150758    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-maas-api-tls\") pod \"maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:57.153258 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.153219    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-maas-api-tls\") pod \"maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:57.193012 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.192973    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:44:57.290274 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.290237    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"]
Jul 28 20:44:57.293919 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.293900    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:44:57.296063 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.296039    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d82230\""
Jul 28 20:44:57.296566 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.296546    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d82230-dockercfg-8gwnz\""
Jul 28 20:44:57.310909 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.310888    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"]
Jul 28 20:44:57.353203 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.353115    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btj6m\" (UniqueName: \"kubernetes.io/projected/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-kube-api-access-btj6m\") pod \"payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw\" (UID: \"9911c937-bd3c-4a7e-a3c7-81f42572e9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:44:57.353203 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.353170    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw\" (UID: \"9911c937-bd3c-4a7e-a3c7-81f42572e9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:44:57.421263 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.421208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:44:57.454748 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.454708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btj6m\" (UniqueName: \"kubernetes.io/projected/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-kube-api-access-btj6m\") pod \"payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw\" (UID: \"9911c937-bd3c-4a7e-a3c7-81f42572e9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:44:57.454903 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.454799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw\" (UID: \"9911c937-bd3c-4a7e-a3c7-81f42572e9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:44:57.455589 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.455569    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw\" (UID: \"9911c937-bd3c-4a7e-a3c7-81f42572e9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:44:57.465788 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.465759    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btj6m\" (UniqueName: \"kubernetes.io/projected/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-kube-api-access-btj6m\") pod \"payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw\" (UID: \"9911c937-bd3c-4a7e-a3c7-81f42572e9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:44:57.590424 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.590403    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"]
Jul 28 20:44:57.592719 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:44:57.592690    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc475ec2e_3a7a_4d54_80f0_71a4379b1fdd.slice/crio-f8ce54b27b3bcaee085c009b1e2512deb96f3e36a4877411e7a2b76f5efa512c WatchSource:0}: Error finding container f8ce54b27b3bcaee085c009b1e2512deb96f3e36a4877411e7a2b76f5efa512c: Status 404 returned error can't find the container with id f8ce54b27b3bcaee085c009b1e2512deb96f3e36a4877411e7a2b76f5efa512c
Jul 28 20:44:57.604550 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.604485    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:44:57.744320 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.744272    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"]
Jul 28 20:44:57.745711 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:44:57.745685    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9911c937_bd3c_4a7e_a3c7_81f42572e9e5.slice/crio-825e3f78b95aba3fa554b308c158550ff47c6cd8d85459032eb91c6d1209ce7d WatchSource:0}: Error finding container 825e3f78b95aba3fa554b308c158550ff47c6cd8d85459032eb91c6d1209ce7d: Status 404 returned error can't find the container with id 825e3f78b95aba3fa554b308c158550ff47c6cd8d85459032eb91c6d1209ce7d
Jul 28 20:44:57.931960 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.931914    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw" event={"ID":"9911c937-bd3c-4a7e-a3c7-81f42572e9e5","Type":"ContainerStarted","Data":"d31966d4294ab808f0d2c81eee08d04c60cf727195c5ffffdea977548e1bf4b5"}
Jul 28 20:44:57.932139 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.931987    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw" event={"ID":"9911c937-bd3c-4a7e-a3c7-81f42572e9e5","Type":"ContainerStarted","Data":"825e3f78b95aba3fa554b308c158550ff47c6cd8d85459032eb91c6d1209ce7d"}
Jul 28 20:44:57.932352 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.932299    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:44:57.933643 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.933611    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs" event={"ID":"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd","Type":"ContainerStarted","Data":"f8ce54b27b3bcaee085c009b1e2512deb96f3e36a4877411e7a2b76f5efa512c"}
Jul 28 20:44:57.973486 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:57.973429    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw" podStartSLOduration=0.973411505 podStartE2EDuration="973.411505ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:44:57.971988307 +0000 UTC m=+2197.586225440" watchObservedRunningTime="2026-07-28 20:44:57.973411505 +0000 UTC m=+2197.587648638"
Jul 28 20:44:58.940377 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:58.940335    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs" event={"ID":"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd","Type":"ContainerStarted","Data":"40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958"}
Jul 28 20:44:58.959740 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:58.959690    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs" podStartSLOduration=2.235866969 podStartE2EDuration="2.959674549s" totalImagesPullingTime="723.80758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:44:58.957803967 +0000 UTC m=+2198.572041099" watchObservedRunningTime="2026-07-28 20:44:58.959674549 +0000 UTC m=+2198.573911682"
Jul 28 20:44:59.945704 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:44:59.945666    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:45:05.955916 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:05.955885    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:45:06.993021 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:06.992988    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:08.943328 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:08.943294    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:45:21.330382 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:21.330350    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:21.668170 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:21.668140    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:22.014812 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:22.014726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:22.542144 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:22.542110    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:25.410416 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.410378    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:25.506064 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.506032    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:25.608097 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.608065    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"]
Jul 28 20:45:25.611657 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.611633    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:25.613935 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.613917    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a7eeff-serving-cert\""
Jul 28 20:45:25.624934 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.624911    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"]
Jul 28 20:45:25.703595 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.703499    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xlld\" (UniqueName: \"kubernetes.io/projected/d8905261-32e5-44dc-bde3-f2f37076c288-kube-api-access-7xlld\") pod \"maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:25.703595 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.703553    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8905261-32e5-44dc-bde3-f2f37076c288-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:25.703798 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.703665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8905261-32e5-44dc-bde3-f2f37076c288-maas-api-tls\") pod \"maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:25.804408 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.804371    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8905261-32e5-44dc-bde3-f2f37076c288-maas-api-tls\") pod \"maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:25.804607 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.804436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xlld\" (UniqueName: \"kubernetes.io/projected/d8905261-32e5-44dc-bde3-f2f37076c288-kube-api-access-7xlld\") pod \"maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:25.804607 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.804478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8905261-32e5-44dc-bde3-f2f37076c288-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:25.804607 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:45:25.804535    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-serving-cert: secret "maas-api-e2e-inf-b-a7eeff-serving-cert" not found
Jul 28 20:45:25.804787 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:45:25.804617    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8905261-32e5-44dc-bde3-f2f37076c288-maas-api-tls podName:d8905261-32e5-44dc-bde3-f2f37076c288 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:26.304598111 +0000 UTC m=+2225.918835223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8905261-32e5-44dc-bde3-f2f37076c288-maas-api-tls") pod "maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz" (UID: "d8905261-32e5-44dc-bde3-f2f37076c288") : secret "maas-api-e2e-inf-b-a7eeff-serving-cert" not found
Jul 28 20:45:25.805216 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.805192    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8905261-32e5-44dc-bde3-f2f37076c288-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:25.814817 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:25.814787    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xlld\" (UniqueName: \"kubernetes.io/projected/d8905261-32e5-44dc-bde3-f2f37076c288-kube-api-access-7xlld\") pod \"maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:26.308935 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.308892    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8905261-32e5-44dc-bde3-f2f37076c288-maas-api-tls\") pod \"maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:26.311457 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.311423    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8905261-32e5-44dc-bde3-f2f37076c288-maas-api-tls\") pod \"maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:26.513452 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.513416    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"]
Jul 28 20:45:26.516943 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.516927    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:45:26.520867 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.520840    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a7eeff-dockercfg-hz6st\""
Jul 28 20:45:26.521032 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.520966    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a7eeff\""
Jul 28 20:45:26.523337 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.523316    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:26.532301 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.532277    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"]
Jul 28 20:45:26.611972 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.611939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zcp\" (UniqueName: \"kubernetes.io/projected/81830d22-7cb2-4087-8431-c9109d123bbe-kube-api-access-w6zcp\") pod \"payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2\" (UID: \"81830d22-7cb2-4087-8431-c9109d123bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:45:26.612123 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.611985    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81830d22-7cb2-4087-8431-c9109d123bbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2\" (UID: \"81830d22-7cb2-4087-8431-c9109d123bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:45:26.671952 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.671851    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"]
Jul 28 20:45:26.674176 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:45:26.674151    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8905261_32e5_44dc_bde3_f2f37076c288.slice/crio-6c40009f094fb7f2c62f8a08e904c854be42dab647f58265e5ff501f71be564e WatchSource:0}: Error finding container 6c40009f094fb7f2c62f8a08e904c854be42dab647f58265e5ff501f71be564e: Status 404 returned error can't find the container with id 6c40009f094fb7f2c62f8a08e904c854be42dab647f58265e5ff501f71be564e
Jul 28 20:45:26.712660 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.712629    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zcp\" (UniqueName: \"kubernetes.io/projected/81830d22-7cb2-4087-8431-c9109d123bbe-kube-api-access-w6zcp\") pod \"payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2\" (UID: \"81830d22-7cb2-4087-8431-c9109d123bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:45:26.712839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.712669    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81830d22-7cb2-4087-8431-c9109d123bbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2\" (UID: \"81830d22-7cb2-4087-8431-c9109d123bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:45:26.713392 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.713375    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81830d22-7cb2-4087-8431-c9109d123bbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2\" (UID: \"81830d22-7cb2-4087-8431-c9109d123bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:45:26.721094 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.721075    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zcp\" (UniqueName: \"kubernetes.io/projected/81830d22-7cb2-4087-8431-c9109d123bbe-kube-api-access-w6zcp\") pod \"payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2\" (UID: \"81830d22-7cb2-4087-8431-c9109d123bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:45:26.795438 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.795400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:26.827805 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.827728    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:45:26.898402 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.898351    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:45:26.971403 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:26.971379    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"]
Jul 28 20:45:26.972937 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:45:26.972902    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81830d22_7cb2_4087_8431_c9109d123bbe.slice/crio-3646b5788328b8f65f1269a52ce14c87888e8b4ec232c110ac6740aed115fa49 WatchSource:0}: Error finding container 3646b5788328b8f65f1269a52ce14c87888e8b4ec232c110ac6740aed115fa49: Status 404 returned error can't find the container with id 3646b5788328b8f65f1269a52ce14c87888e8b4ec232c110ac6740aed115fa49
Jul 28 20:45:27.073285 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:27.073248    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2" event={"ID":"81830d22-7cb2-4087-8431-c9109d123bbe","Type":"ContainerStarted","Data":"b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb"}
Jul 28 20:45:27.073546 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:27.073343    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2" event={"ID":"81830d22-7cb2-4087-8431-c9109d123bbe","Type":"ContainerStarted","Data":"3646b5788328b8f65f1269a52ce14c87888e8b4ec232c110ac6740aed115fa49"}
Jul 28 20:45:27.073546 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:27.073373    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:45:27.074379 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:27.074351    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz" event={"ID":"d8905261-32e5-44dc-bde3-f2f37076c288","Type":"ContainerStarted","Data":"6c40009f094fb7f2c62f8a08e904c854be42dab647f58265e5ff501f71be564e"}
Jul 28 20:45:27.100370 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:27.100276    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2" podStartSLOduration=1.100260339 podStartE2EDuration="1.100260339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:27.09908078 +0000 UTC m=+2226.713317926" watchObservedRunningTime="2026-07-28 20:45:27.100260339 +0000 UTC m=+2226.714497469"
Jul 28 20:45:28.081421 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:28.081381    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz" event={"ID":"d8905261-32e5-44dc-bde3-f2f37076c288","Type":"ContainerStarted","Data":"817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973"}
Jul 28 20:45:28.081902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:28.081677    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:28.101622 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:28.101573    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz" podStartSLOduration=2.397550443 podStartE2EDuration="3.101557965s" totalImagesPullingTime="704.007522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:28.099866796 +0000 UTC m=+2227.714103930" watchObservedRunningTime="2026-07-28 20:45:28.101557965 +0000 UTC m=+2227.715795177"
Jul 28 20:45:34.091704 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:34.091673    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:45:38.083572 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:38.083538    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:45:50.441491 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:45:50.441375    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:46:05.455415 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:46:05.455361    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:46:27.039163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:46:27.039109    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:15.502921 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:15.502828    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:16.413738 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:16.413673    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:27.298043 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.298006    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"]
Jul 28 20:47:27.298533 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.298256    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs" podUID="c475ec2e-3a7a-4d54-80f0-71a4379b1fdd" containerName="maas-api" containerID="cri-o://40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958" gracePeriod=30
Jul 28 20:47:27.330833 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.330800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:27.377672 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.377638    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"]
Jul 28 20:47:27.377839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.377819    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw" podUID="9911c937-bd3c-4a7e-a3c7-81f42572e9e5" containerName="payload-pre-processing" containerID="cri-o://d31966d4294ab808f0d2c81eee08d04c60cf727195c5ffffdea977548e1bf4b5" gracePeriod=30
Jul 28 20:47:27.452954 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.452925    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:27.584868 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.584829    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:27.626053 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.626026    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:47:27.659853 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.659825    2575 generic.go:356] "Generic (PLEG): container finished" podID="9911c937-bd3c-4a7e-a3c7-81f42572e9e5" containerID="d31966d4294ab808f0d2c81eee08d04c60cf727195c5ffffdea977548e1bf4b5" exitCode=0
Jul 28 20:47:27.659966 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.659892    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw" event={"ID":"9911c937-bd3c-4a7e-a3c7-81f42572e9e5","Type":"ContainerDied","Data":"d31966d4294ab808f0d2c81eee08d04c60cf727195c5ffffdea977548e1bf4b5"}
Jul 28 20:47:27.659966 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.659959    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw" event={"ID":"9911c937-bd3c-4a7e-a3c7-81f42572e9e5","Type":"ContainerDied","Data":"825e3f78b95aba3fa554b308c158550ff47c6cd8d85459032eb91c6d1209ce7d"}
Jul 28 20:47:27.660090 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.659986    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="825e3f78b95aba3fa554b308c158550ff47c6cd8d85459032eb91c6d1209ce7d"
Jul 28 20:47:27.660523 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.660505    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:47:27.661290 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.661270    2575 generic.go:356] "Generic (PLEG): container finished" podID="c475ec2e-3a7a-4d54-80f0-71a4379b1fdd" containerID="40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958" exitCode=0
Jul 28 20:47:27.661411 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.661309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs" event={"ID":"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd","Type":"ContainerDied","Data":"40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958"}
Jul 28 20:47:27.661411 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.661344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs" event={"ID":"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd","Type":"ContainerDied","Data":"f8ce54b27b3bcaee085c009b1e2512deb96f3e36a4877411e7a2b76f5efa512c"}
Jul 28 20:47:27.661411 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.661365    2575 scope.go:117] "RemoveContainer" containerID="40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958"
Jul 28 20:47:27.661411 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.661311    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"
Jul 28 20:47:27.682598 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.682575    2575 scope.go:117] "RemoveContainer" containerID="40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958"
Jul 28 20:47:27.682895 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:47:27.682873    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958\": container with ID starting with 40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958 not found: ID does not exist" containerID="40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958"
Jul 28 20:47:27.682985 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.682900    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958"} err="failed to get container status \"40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958\": rpc error: code = NotFound desc = could not find container \"40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958\": container with ID starting with 40ae935a5018e979d54761bafe30406e34afc1c0767b5dbb51517c697cfb0958 not found: ID does not exist"
Jul 28 20:47:27.705514 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.705480    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:27.726914 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.726882    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-maas-api-tls\") pod \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") "
Jul 28 20:47:27.727054 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.726931    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-plugins-config-volume\") pod \"9911c937-bd3c-4a7e-a3c7-81f42572e9e5\" (UID: \"9911c937-bd3c-4a7e-a3c7-81f42572e9e5\") "
Jul 28 20:47:27.727054 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.726997    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-openshift-service-ca\") pod \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") "
Jul 28 20:47:27.727054 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.727013    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpxql\" (UniqueName: \"kubernetes.io/projected/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-kube-api-access-mpxql\") pod \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\" (UID: \"c475ec2e-3a7a-4d54-80f0-71a4379b1fdd\") "
Jul 28 20:47:27.727054 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.727028    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btj6m\" (UniqueName: \"kubernetes.io/projected/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-kube-api-access-btj6m\") pod \"9911c937-bd3c-4a7e-a3c7-81f42572e9e5\" (UID: \"9911c937-bd3c-4a7e-a3c7-81f42572e9e5\") "
Jul 28 20:47:27.727431 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.727392    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9911c937-bd3c-4a7e-a3c7-81f42572e9e5" (UID: "9911c937-bd3c-4a7e-a3c7-81f42572e9e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:27.727431 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.727406    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c475ec2e-3a7a-4d54-80f0-71a4379b1fdd" (UID: "c475ec2e-3a7a-4d54-80f0-71a4379b1fdd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:27.729404 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.729361    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-kube-api-access-mpxql" (OuterVolumeSpecName: "kube-api-access-mpxql") pod "c475ec2e-3a7a-4d54-80f0-71a4379b1fdd" (UID: "c475ec2e-3a7a-4d54-80f0-71a4379b1fdd"). InnerVolumeSpecName "kube-api-access-mpxql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:47:27.729777 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.729740    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-kube-api-access-btj6m" (OuterVolumeSpecName: "kube-api-access-btj6m") pod "9911c937-bd3c-4a7e-a3c7-81f42572e9e5" (UID: "9911c937-bd3c-4a7e-a3c7-81f42572e9e5"). InnerVolumeSpecName "kube-api-access-btj6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:47:27.730294 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.730269    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c475ec2e-3a7a-4d54-80f0-71a4379b1fdd" (UID: "c475ec2e-3a7a-4d54-80f0-71a4379b1fdd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:27.828160 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.828126    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:47:27.828160 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.828158    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpxql\" (UniqueName: \"kubernetes.io/projected/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-kube-api-access-mpxql\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:47:27.828401 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.828170    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btj6m\" (UniqueName: \"kubernetes.io/projected/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-kube-api-access-btj6m\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:47:27.828401 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.828183    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:47:27.828401 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.828195    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9911c937-bd3c-4a7e-a3c7-81f42572e9e5-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:47:27.994657 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.994625    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"]
Jul 28 20:47:27.999094 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:27.999071    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d82230-645fccb54b-jp6bs"]
Jul 28 20:47:28.125465 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:28.125429    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:28.667219 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:28.667193    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"
Jul 28 20:47:28.695035 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:28.695009    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"]
Jul 28 20:47:28.698482 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:28.698458    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d82230-5c9896bb84-m4gxw"]
Jul 28 20:47:28.836169 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:28.836137    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9911c937-bd3c-4a7e-a3c7-81f42572e9e5" path="/var/lib/kubelet/pods/9911c937-bd3c-4a7e-a3c7-81f42572e9e5/volumes"
Jul 28 20:47:28.836564 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:28.836549    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c475ec2e-3a7a-4d54-80f0-71a4379b1fdd" path="/var/lib/kubelet/pods/c475ec2e-3a7a-4d54-80f0-71a4379b1fdd/volumes"
Jul 28 20:47:34.614308 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:34.614274    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:34.699133 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:34.699107    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:34.897846 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:34.897817    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:35.145655 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:35.145625    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:45.807589 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:45.807550    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"]
Jul 28 20:47:45.808113 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:45.807801    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz" podUID="d8905261-32e5-44dc-bde3-f2f37076c288" containerName="maas-api" containerID="cri-o://817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973" gracePeriod=30
Jul 28 20:47:45.855960 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:45.855924    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:45.911426 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:45.911391    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"]
Jul 28 20:47:45.911654 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:45.911615    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2" podUID="81830d22-7cb2-4087-8431-c9109d123bbe" containerName="payload-pre-processing" containerID="cri-o://b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb" gracePeriod=30
Jul 28 20:47:45.962780 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:45.962752    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:46.061624 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.061551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:46.073484 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.073460    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:47:46.158597 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.158564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:46.181523 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.181502    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:47:46.196655 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.196633    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xlld\" (UniqueName: \"kubernetes.io/projected/d8905261-32e5-44dc-bde3-f2f37076c288-kube-api-access-7xlld\") pod \"d8905261-32e5-44dc-bde3-f2f37076c288\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") "
Jul 28 20:47:46.196753 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.196671    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8905261-32e5-44dc-bde3-f2f37076c288-maas-api-tls\") pod \"d8905261-32e5-44dc-bde3-f2f37076c288\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") "
Jul 28 20:47:46.196802 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.196771    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8905261-32e5-44dc-bde3-f2f37076c288-openshift-service-ca\") pod \"d8905261-32e5-44dc-bde3-f2f37076c288\" (UID: \"d8905261-32e5-44dc-bde3-f2f37076c288\") "
Jul 28 20:47:46.197198 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.197167    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8905261-32e5-44dc-bde3-f2f37076c288-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8905261-32e5-44dc-bde3-f2f37076c288" (UID: "d8905261-32e5-44dc-bde3-f2f37076c288"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:46.198802 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.198778    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8905261-32e5-44dc-bde3-f2f37076c288-kube-api-access-7xlld" (OuterVolumeSpecName: "kube-api-access-7xlld") pod "d8905261-32e5-44dc-bde3-f2f37076c288" (UID: "d8905261-32e5-44dc-bde3-f2f37076c288"). InnerVolumeSpecName "kube-api-access-7xlld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:47:46.198891 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.198833    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8905261-32e5-44dc-bde3-f2f37076c288-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8905261-32e5-44dc-bde3-f2f37076c288" (UID: "d8905261-32e5-44dc-bde3-f2f37076c288"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:46.297790 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.297761    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81830d22-7cb2-4087-8431-c9109d123bbe-plugins-config-volume\") pod \"81830d22-7cb2-4087-8431-c9109d123bbe\" (UID: \"81830d22-7cb2-4087-8431-c9109d123bbe\") "
Jul 28 20:47:46.298067 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.297812    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6zcp\" (UniqueName: \"kubernetes.io/projected/81830d22-7cb2-4087-8431-c9109d123bbe-kube-api-access-w6zcp\") pod \"81830d22-7cb2-4087-8431-c9109d123bbe\" (UID: \"81830d22-7cb2-4087-8431-c9109d123bbe\") "
Jul 28 20:47:46.298067 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.297959    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8905261-32e5-44dc-bde3-f2f37076c288-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:47:46.298067 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.297973    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xlld\" (UniqueName: \"kubernetes.io/projected/d8905261-32e5-44dc-bde3-f2f37076c288-kube-api-access-7xlld\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:47:46.298067 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.297982    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8905261-32e5-44dc-bde3-f2f37076c288-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:47:46.298299 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.298103    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81830d22-7cb2-4087-8431-c9109d123bbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81830d22-7cb2-4087-8431-c9109d123bbe" (UID: "81830d22-7cb2-4087-8431-c9109d123bbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:46.299866 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.299844    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81830d22-7cb2-4087-8431-c9109d123bbe-kube-api-access-w6zcp" (OuterVolumeSpecName: "kube-api-access-w6zcp") pod "81830d22-7cb2-4087-8431-c9109d123bbe" (UID: "81830d22-7cb2-4087-8431-c9109d123bbe"). InnerVolumeSpecName "kube-api-access-w6zcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:47:46.398930 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.398901    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6zcp\" (UniqueName: \"kubernetes.io/projected/81830d22-7cb2-4087-8431-c9109d123bbe-kube-api-access-w6zcp\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:47:46.398930 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.398931    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81830d22-7cb2-4087-8431-c9109d123bbe-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:47:46.511509 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.511477    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:46.753350 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.753266    2575 generic.go:356] "Generic (PLEG): container finished" podID="81830d22-7cb2-4087-8431-c9109d123bbe" containerID="b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb" exitCode=0
Jul 28 20:47:46.753350 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.753308    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"
Jul 28 20:47:46.753554 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.753356    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2" event={"ID":"81830d22-7cb2-4087-8431-c9109d123bbe","Type":"ContainerDied","Data":"b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb"}
Jul 28 20:47:46.753554 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.753416    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2" event={"ID":"81830d22-7cb2-4087-8431-c9109d123bbe","Type":"ContainerDied","Data":"3646b5788328b8f65f1269a52ce14c87888e8b4ec232c110ac6740aed115fa49"}
Jul 28 20:47:46.753554 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.753451    2575 scope.go:117] "RemoveContainer" containerID="b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb"
Jul 28 20:47:46.754713 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.754692    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8905261-32e5-44dc-bde3-f2f37076c288" containerID="817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973" exitCode=0
Jul 28 20:47:46.754822 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.754740    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"
Jul 28 20:47:46.754822 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.754770    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz" event={"ID":"d8905261-32e5-44dc-bde3-f2f37076c288","Type":"ContainerDied","Data":"817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973"}
Jul 28 20:47:46.754822 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.754812    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz" event={"ID":"d8905261-32e5-44dc-bde3-f2f37076c288","Type":"ContainerDied","Data":"6c40009f094fb7f2c62f8a08e904c854be42dab647f58265e5ff501f71be564e"}
Jul 28 20:47:46.764379 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.764358    2575 scope.go:117] "RemoveContainer" containerID="b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb"
Jul 28 20:47:46.764659 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:47:46.764640    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb\": container with ID starting with b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb not found: ID does not exist" containerID="b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb"
Jul 28 20:47:46.764737 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.764669    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb"} err="failed to get container status \"b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb\": rpc error: code = NotFound desc = could not find container \"b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb\": container with ID starting with b5afc7d101cabfce4049dda6ccb4d51d44eb687e70c14650275cf888b7174ebb not found: ID does not exist"
Jul 28 20:47:46.764737 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.764684    2575 scope.go:117] "RemoveContainer" containerID="817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973"
Jul 28 20:47:46.772601 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.772576    2575 scope.go:117] "RemoveContainer" containerID="817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973"
Jul 28 20:47:46.772874 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:47:46.772848    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973\": container with ID starting with 817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973 not found: ID does not exist" containerID="817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973"
Jul 28 20:47:46.772923 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.772881    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973"} err="failed to get container status \"817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973\": rpc error: code = NotFound desc = could not find container \"817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973\": container with ID starting with 817f9351f74fc8972a481d435d907860301e5ad4daf33660c05a34217040b973 not found: ID does not exist"
Jul 28 20:47:46.785955 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.785928    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"]
Jul 28 20:47:46.793088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.793053    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a7eeff-5bf9dd9648-6wsg2"]
Jul 28 20:47:46.807250 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.807211    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"]
Jul 28 20:47:46.820371 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.820348    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7eeff-7cd6b8d8cb-q42xz"]
Jul 28 20:47:46.836530 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.836503    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81830d22-7cb2-4087-8431-c9109d123bbe" path="/var/lib/kubelet/pods/81830d22-7cb2-4087-8431-c9109d123bbe/volumes"
Jul 28 20:47:46.836904 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:46.836891    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8905261-32e5-44dc-bde3-f2f37076c288" path="/var/lib/kubelet/pods/d8905261-32e5-44dc-bde3-f2f37076c288/volumes"
Jul 28 20:47:52.858936 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:52.858904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:52.942971 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:52.942935    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:53.180861 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:53.180820    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:53.400577 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:53.400542    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:53.788159 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:53.788130    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:54.165924 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:54.165890    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:54.369666 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:54.369633    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:57.446135 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.446102    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:57.558456 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558422    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"]
Jul 28 20:47:57.558831 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558817    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9911c937-bd3c-4a7e-a3c7-81f42572e9e5" containerName="payload-pre-processing"
Jul 28 20:47:57.558902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558833    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9911c937-bd3c-4a7e-a3c7-81f42572e9e5" containerName="payload-pre-processing"
Jul 28 20:47:57.558902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558847    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c475ec2e-3a7a-4d54-80f0-71a4379b1fdd" containerName="maas-api"
Jul 28 20:47:57.558902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558857    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c475ec2e-3a7a-4d54-80f0-71a4379b1fdd" containerName="maas-api"
Jul 28 20:47:57.558902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558868    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8905261-32e5-44dc-bde3-f2f37076c288" containerName="maas-api"
Jul 28 20:47:57.558902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558873    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8905261-32e5-44dc-bde3-f2f37076c288" containerName="maas-api"
Jul 28 20:47:57.558902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558883    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81830d22-7cb2-4087-8431-c9109d123bbe" containerName="payload-pre-processing"
Jul 28 20:47:57.558902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558889    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="81830d22-7cb2-4087-8431-c9109d123bbe" containerName="payload-pre-processing"
Jul 28 20:47:57.559279 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558950    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c475ec2e-3a7a-4d54-80f0-71a4379b1fdd" containerName="maas-api"
Jul 28 20:47:57.559279 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558961    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8905261-32e5-44dc-bde3-f2f37076c288" containerName="maas-api"
Jul 28 20:47:57.559279 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558967    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9911c937-bd3c-4a7e-a3c7-81f42572e9e5" containerName="payload-pre-processing"
Jul 28 20:47:57.559279 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.558973    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="81830d22-7cb2-4087-8431-c9109d123bbe" containerName="payload-pre-processing"
Jul 28 20:47:57.562007 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.561989    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:57.565562 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.565537    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-65eee9-serving-cert\""
Jul 28 20:47:57.567722 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.567695    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:57.585070 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.585042    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"]
Jul 28 20:47:57.595850 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.595815    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-openshift-service-ca\") pod \"maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:57.595990 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.595880    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpd7j\" (UniqueName: \"kubernetes.io/projected/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-kube-api-access-cpd7j\") pod \"maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:57.595990 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.595962    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-maas-api-tls\") pod \"maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:57.695721 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.695689    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:57.696879 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.696819    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-openshift-service-ca\") pod \"maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:57.696962 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.696889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpd7j\" (UniqueName: \"kubernetes.io/projected/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-kube-api-access-cpd7j\") pod \"maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:57.696962 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.696928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-maas-api-tls\") pod \"maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:57.697068 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:47:57.697041    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-serving-cert: secret "maas-api-e2e-api-a-65eee9-serving-cert" not found
Jul 28 20:47:57.697121 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:47:57.697096    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-maas-api-tls podName:39f6b6c4-57c8-4938-892f-7ba0fa3ac87d nodeName:}" failed. No retries permitted until 2026-07-28 20:47:58.197076487 +0000 UTC m=+2377.811313599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-maas-api-tls") pod "maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6" (UID: "39f6b6c4-57c8-4938-892f-7ba0fa3ac87d") : secret "maas-api-e2e-api-a-65eee9-serving-cert" not found
Jul 28 20:47:57.697445 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.697425    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-openshift-service-ca\") pod \"maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:57.721344 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:57.721314    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpd7j\" (UniqueName: \"kubernetes.io/projected/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-kube-api-access-cpd7j\") pod \"maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:58.023333 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.023251    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:58.202285 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.202249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-maas-api-tls\") pod \"maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:58.204710 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.204684    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-maas-api-tls\") pod \"maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:58.304886 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.304799    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:58.440007 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.439977    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:47:58.472198 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.472169    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:58.528407 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.528372    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"]
Jul 28 20:47:58.537587 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.534451    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:47:58.556676 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.556603    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-65eee9\""
Jul 28 20:47:58.557392 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.557362    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-65eee9-dockercfg-7ldmd\""
Jul 28 20:47:58.575902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.575802    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"]
Jul 28 20:47:58.605498 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.605468    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724cbffa-8c0a-41f0-9173-8499723f819e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7\" (UID: \"724cbffa-8c0a-41f0-9173-8499723f819e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:47:58.605650 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.605590    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tqxg\" (UniqueName: \"kubernetes.io/projected/724cbffa-8c0a-41f0-9173-8499723f819e-kube-api-access-7tqxg\") pod \"payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7\" (UID: \"724cbffa-8c0a-41f0-9173-8499723f819e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:47:58.636011 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.635990    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"]
Jul 28 20:47:58.638385 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:47:58.638360    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f6b6c4_57c8_4938_892f_7ba0fa3ac87d.slice/crio-25ccfc9ec748e493ba017c9a3927817b236daafb6d13c8ad81e584c6c6992b1d WatchSource:0}: Error finding container 25ccfc9ec748e493ba017c9a3927817b236daafb6d13c8ad81e584c6c6992b1d: Status 404 returned error can't find the container with id 25ccfc9ec748e493ba017c9a3927817b236daafb6d13c8ad81e584c6c6992b1d
Jul 28 20:47:58.639731 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.639712    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:47:58.707071 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.707035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724cbffa-8c0a-41f0-9173-8499723f819e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7\" (UID: \"724cbffa-8c0a-41f0-9173-8499723f819e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:47:58.707252 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.707106    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tqxg\" (UniqueName: \"kubernetes.io/projected/724cbffa-8c0a-41f0-9173-8499723f819e-kube-api-access-7tqxg\") pod \"payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7\" (UID: \"724cbffa-8c0a-41f0-9173-8499723f819e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:47:58.707819 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.707797    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724cbffa-8c0a-41f0-9173-8499723f819e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7\" (UID: \"724cbffa-8c0a-41f0-9173-8499723f819e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:47:58.717706 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.717677    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tqxg\" (UniqueName: \"kubernetes.io/projected/724cbffa-8c0a-41f0-9173-8499723f819e-kube-api-access-7tqxg\") pod \"payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7\" (UID: \"724cbffa-8c0a-41f0-9173-8499723f819e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:47:58.817646 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.817562    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6" event={"ID":"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d","Type":"ContainerStarted","Data":"25ccfc9ec748e493ba017c9a3927817b236daafb6d13c8ad81e584c6c6992b1d"}
Jul 28 20:47:58.848023 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.847992    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:47:58.993191 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:58.993160    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"]
Jul 28 20:47:58.995795 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:47:58.995758    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724cbffa_8c0a_41f0_9173_8499723f819e.slice/crio-f6dc5dd9967d4c62ee1f12a653dbc32257377c61c96624862d3c2d355d76b5e7 WatchSource:0}: Error finding container f6dc5dd9967d4c62ee1f12a653dbc32257377c61c96624862d3c2d355d76b5e7: Status 404 returned error can't find the container with id f6dc5dd9967d4c62ee1f12a653dbc32257377c61c96624862d3c2d355d76b5e7
Jul 28 20:47:59.829276 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:59.829208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7" event={"ID":"724cbffa-8c0a-41f0-9173-8499723f819e","Type":"ContainerStarted","Data":"8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290"}
Jul 28 20:47:59.829779 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:59.829334    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7" event={"ID":"724cbffa-8c0a-41f0-9173-8499723f819e","Type":"ContainerStarted","Data":"f6dc5dd9967d4c62ee1f12a653dbc32257377c61c96624862d3c2d355d76b5e7"}
Jul 28 20:47:59.829779 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:59.829525    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:47:59.831333 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:59.831309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6" event={"ID":"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d","Type":"ContainerStarted","Data":"5bdae5cb56cb9187e0afbaa54613309b8493403a5bc4423574f54573597ea0fd"}
Jul 28 20:47:59.831453 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:59.831440    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:47:59.855785 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:59.855725    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7" podStartSLOduration=1.855708937 podStartE2EDuration="1.855708937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:59.85314428 +0000 UTC m=+2379.467381414" watchObservedRunningTime="2026-07-28 20:47:59.855708937 +0000 UTC m=+2379.469946071"
Jul 28 20:47:59.876498 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:47:59.876450    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6" podStartSLOduration=2.468483683 podStartE2EDuration="2.87643154s" totalImagesPullingTime="407.947857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:59.875578036 +0000 UTC m=+2379.489815171" watchObservedRunningTime="2026-07-28 20:47:59.87643154 +0000 UTC m=+2379.490668672"
Jul 28 20:48:05.842674 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:05.842642    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:48:10.837216 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:10.837189    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:48:22.526943 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:22.526910    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:23.091149 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:23.091121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:23.569857 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:23.569817    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:23.754668 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:23.754638    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:26.954277 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:26.954222    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:27.244929 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:27.244835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:28.207642 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.207606    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"]
Jul 28 20:48:28.211091 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.211075    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.216646 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.216627    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-42ff09-serving-cert\""
Jul 28 20:48:28.265050 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.265014    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"]
Jul 28 20:48:28.288815 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.288785    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:28.375064 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.375031    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh6cr\" (UniqueName: \"kubernetes.io/projected/e84739c8-1edf-4a5f-92af-d5266a8d9b92-kube-api-access-nh6cr\") pod \"maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.375064 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.375067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e84739c8-1edf-4a5f-92af-d5266a8d9b92-openshift-service-ca\") pod \"maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.375336 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.375096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e84739c8-1edf-4a5f-92af-d5266a8d9b92-maas-api-tls\") pod \"maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.476292 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.476193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6cr\" (UniqueName: \"kubernetes.io/projected/e84739c8-1edf-4a5f-92af-d5266a8d9b92-kube-api-access-nh6cr\") pod \"maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.476292 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.476260    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e84739c8-1edf-4a5f-92af-d5266a8d9b92-openshift-service-ca\") pod \"maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.476474 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.476306    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e84739c8-1edf-4a5f-92af-d5266a8d9b92-maas-api-tls\") pod \"maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.476905 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.476884    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e84739c8-1edf-4a5f-92af-d5266a8d9b92-openshift-service-ca\") pod \"maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.478722 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.478702    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e84739c8-1edf-4a5f-92af-d5266a8d9b92-maas-api-tls\") pod \"maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.486804 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.486783    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6cr\" (UniqueName: \"kubernetes.io/projected/e84739c8-1edf-4a5f-92af-d5266a8d9b92-kube-api-access-nh6cr\") pod \"maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.522692 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.522666    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:28.558322 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.558289    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:28.676943 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.676913    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"]
Jul 28 20:48:28.678439 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:48:28.678407    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84739c8_1edf_4a5f_92af_d5266a8d9b92.slice/crio-772f6d7e70978e28930c1b2b849bbae23ea7383df20f6d255545d9c42858bca8 WatchSource:0}: Error finding container 772f6d7e70978e28930c1b2b849bbae23ea7383df20f6d255545d9c42858bca8: Status 404 returned error can't find the container with id 772f6d7e70978e28930c1b2b849bbae23ea7383df20f6d255545d9c42858bca8
Jul 28 20:48:28.977337 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:28.977301    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7" event={"ID":"e84739c8-1edf-4a5f-92af-d5266a8d9b92","Type":"ContainerStarted","Data":"772f6d7e70978e28930c1b2b849bbae23ea7383df20f6d255545d9c42858bca8"}
Jul 28 20:48:29.192515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.192483    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:29.427956 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.427917    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:29.582968 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.582936    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"]
Jul 28 20:48:29.586596 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.586577    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:48:29.590148 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.590127    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-42ff09-dockercfg-8mm4f\""
Jul 28 20:48:29.596924 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.596904    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-42ff09\""
Jul 28 20:48:29.610572 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.610549    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"]
Jul 28 20:48:29.687796 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.687724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj\" (UID: \"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:48:29.687796 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.687754    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhc98\" (UniqueName: \"kubernetes.io/projected/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-kube-api-access-nhc98\") pod \"payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj\" (UID: \"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:48:29.789191 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.789156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj\" (UID: \"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:48:29.789191 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.789190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc98\" (UniqueName: \"kubernetes.io/projected/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-kube-api-access-nhc98\") pod \"payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj\" (UID: \"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:48:29.789973 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.789939    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj\" (UID: \"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:48:29.803142 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.803120    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhc98\" (UniqueName: \"kubernetes.io/projected/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-kube-api-access-nhc98\") pod \"payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj\" (UID: \"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:48:29.897211 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.897172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:48:29.984218 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.984184    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7" event={"ID":"e84739c8-1edf-4a5f-92af-d5266a8d9b92","Type":"ContainerStarted","Data":"803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1"}
Jul 28 20:48:29.984434 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.984274    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:29.992019 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:29.991993    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:30.013517 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:30.013391    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7" podStartSLOduration=1.633320779 podStartE2EDuration="2.013375093s" totalImagesPullingTime="380.054314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:30.010555191 +0000 UTC m=+2409.624792325" watchObservedRunningTime="2026-07-28 20:48:30.013375093 +0000 UTC m=+2409.627612226"
Jul 28 20:48:30.051707 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:30.051678    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"]
Jul 28 20:48:30.055215 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:48:30.055191    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8a6a62_b358_40d8_9bd2_31465bb8f3ab.slice/crio-5f1ed1e3eaa4523124ba6c36638cebf993b950497b671a392e8e8ae4d3720164 WatchSource:0}: Error finding container 5f1ed1e3eaa4523124ba6c36638cebf993b950497b671a392e8e8ae4d3720164: Status 404 returned error can't find the container with id 5f1ed1e3eaa4523124ba6c36638cebf993b950497b671a392e8e8ae4d3720164
Jul 28 20:48:30.991452 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:30.991414    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj" event={"ID":"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab","Type":"ContainerStarted","Data":"7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0"}
Jul 28 20:48:30.991861 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:30.991471    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:48:30.991861 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:30.991494    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj" event={"ID":"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab","Type":"ContainerStarted","Data":"5f1ed1e3eaa4523124ba6c36638cebf993b950497b671a392e8e8ae4d3720164"}
Jul 28 20:48:35.997283 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:35.997247    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:48:36.042154 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:36.042102    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj" podStartSLOduration=7.042087219 podStartE2EDuration="7.042087219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:31.036526704 +0000 UTC m=+2410.650763838" watchObservedRunningTime="2026-07-28 20:48:36.042087219 +0000 UTC m=+2415.656324389"
Jul 28 20:48:41.679344 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:41.679308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:48:42.005328 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:48:42.005222    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:49:05.046443 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.046359    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"]
Jul 28 20:49:05.046982 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.046574    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6" podUID="39f6b6c4-57c8-4938-892f-7ba0fa3ac87d" containerName="maas-api" containerID="cri-o://5bdae5cb56cb9187e0afbaa54613309b8493403a5bc4423574f54573597ea0fd" gracePeriod=30
Jul 28 20:49:05.055804 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.055779    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:05.115414 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.115381    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"]
Jul 28 20:49:05.115667 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.115622    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7" podUID="724cbffa-8c0a-41f0-9173-8499723f819e" containerName="payload-pre-processing" containerID="cri-o://8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290" gracePeriod=30
Jul 28 20:49:05.158897 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.158869    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:05.170075 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.170052    2575 generic.go:356] "Generic (PLEG): container finished" podID="39f6b6c4-57c8-4938-892f-7ba0fa3ac87d" containerID="5bdae5cb56cb9187e0afbaa54613309b8493403a5bc4423574f54573597ea0fd" exitCode=0
Jul 28 20:49:05.170195 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.170122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6" event={"ID":"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d","Type":"ContainerDied","Data":"5bdae5cb56cb9187e0afbaa54613309b8493403a5bc4423574f54573597ea0fd"}
Jul 28 20:49:05.245924 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.245895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:05.346724 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.346687    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:05.375646 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.375618    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:49:05.409333 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.409308    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:49:05.496665 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.496635    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-openshift-service-ca\") pod \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") "
Jul 28 20:49:05.496832 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.496681    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpd7j\" (UniqueName: \"kubernetes.io/projected/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-kube-api-access-cpd7j\") pod \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") "
Jul 28 20:49:05.496832 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.496818    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tqxg\" (UniqueName: \"kubernetes.io/projected/724cbffa-8c0a-41f0-9173-8499723f819e-kube-api-access-7tqxg\") pod \"724cbffa-8c0a-41f0-9173-8499723f819e\" (UID: \"724cbffa-8c0a-41f0-9173-8499723f819e\") "
Jul 28 20:49:05.496921 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.496839    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-maas-api-tls\") pod \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\" (UID: \"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d\") "
Jul 28 20:49:05.496921 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.496873    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724cbffa-8c0a-41f0-9173-8499723f819e-plugins-config-volume\") pod \"724cbffa-8c0a-41f0-9173-8499723f819e\" (UID: \"724cbffa-8c0a-41f0-9173-8499723f819e\") "
Jul 28 20:49:05.497088 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.497061    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39f6b6c4-57c8-4938-892f-7ba0fa3ac87d" (UID: "39f6b6c4-57c8-4938-892f-7ba0fa3ac87d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:05.497335 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.497311    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724cbffa-8c0a-41f0-9173-8499723f819e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "724cbffa-8c0a-41f0-9173-8499723f819e" (UID: "724cbffa-8c0a-41f0-9173-8499723f819e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:05.498861 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.498832    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724cbffa-8c0a-41f0-9173-8499723f819e-kube-api-access-7tqxg" (OuterVolumeSpecName: "kube-api-access-7tqxg") pod "724cbffa-8c0a-41f0-9173-8499723f819e" (UID: "724cbffa-8c0a-41f0-9173-8499723f819e"). InnerVolumeSpecName "kube-api-access-7tqxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:49:05.498962 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.498864    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39f6b6c4-57c8-4938-892f-7ba0fa3ac87d" (UID: "39f6b6c4-57c8-4938-892f-7ba0fa3ac87d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:49:05.498962 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.498866    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-kube-api-access-cpd7j" (OuterVolumeSpecName: "kube-api-access-cpd7j") pod "39f6b6c4-57c8-4938-892f-7ba0fa3ac87d" (UID: "39f6b6c4-57c8-4938-892f-7ba0fa3ac87d"). InnerVolumeSpecName "kube-api-access-cpd7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:49:05.597950 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.597867    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tqxg\" (UniqueName: \"kubernetes.io/projected/724cbffa-8c0a-41f0-9173-8499723f819e-kube-api-access-7tqxg\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:49:05.597950 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.597900    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:49:05.597950 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.597916    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724cbffa-8c0a-41f0-9173-8499723f819e-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:49:05.597950 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.597930    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:49:05.597950 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.597945    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpd7j\" (UniqueName: \"kubernetes.io/projected/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d-kube-api-access-cpd7j\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:49:05.937494 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:05.937456    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:06.176237 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.176196    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"
Jul 28 20:49:06.176684 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.176219    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6" event={"ID":"39f6b6c4-57c8-4938-892f-7ba0fa3ac87d","Type":"ContainerDied","Data":"25ccfc9ec748e493ba017c9a3927817b236daafb6d13c8ad81e584c6c6992b1d"}
Jul 28 20:49:06.176684 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.176291    2575 scope.go:117] "RemoveContainer" containerID="5bdae5cb56cb9187e0afbaa54613309b8493403a5bc4423574f54573597ea0fd"
Jul 28 20:49:06.177650 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.177628    2575 generic.go:356] "Generic (PLEG): container finished" podID="724cbffa-8c0a-41f0-9173-8499723f819e" containerID="8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290" exitCode=0
Jul 28 20:49:06.177732 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.177676    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7" event={"ID":"724cbffa-8c0a-41f0-9173-8499723f819e","Type":"ContainerDied","Data":"8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290"}
Jul 28 20:49:06.177732 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.177718    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7" event={"ID":"724cbffa-8c0a-41f0-9173-8499723f819e","Type":"ContainerDied","Data":"f6dc5dd9967d4c62ee1f12a653dbc32257377c61c96624862d3c2d355d76b5e7"}
Jul 28 20:49:06.177835 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.177746    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"
Jul 28 20:49:06.186655 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.186635    2575 scope.go:117] "RemoveContainer" containerID="8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290"
Jul 28 20:49:06.194925 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.194907    2575 scope.go:117] "RemoveContainer" containerID="8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290"
Jul 28 20:49:06.195253 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:49:06.195212    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290\": container with ID starting with 8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290 not found: ID does not exist" containerID="8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290"
Jul 28 20:49:06.195316 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.195261    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290"} err="failed to get container status \"8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290\": rpc error: code = NotFound desc = could not find container \"8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290\": container with ID starting with 8dd4151e38290eb4b819cb1f7f8ff422fb5ae6ec5cadbc9e3aeba113342c0290 not found: ID does not exist"
Jul 28 20:49:06.216788 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.216754    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"]
Jul 28 20:49:06.228712 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.228676    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65eee9-764dfbfbf6-l4wd7"]
Jul 28 20:49:06.258416 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.258385    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"]
Jul 28 20:49:06.272785 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.272758    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-65eee9-7d5b5d87cc-jpdh6"]
Jul 28 20:49:06.836914 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.836882    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39f6b6c4-57c8-4938-892f-7ba0fa3ac87d" path="/var/lib/kubelet/pods/39f6b6c4-57c8-4938-892f-7ba0fa3ac87d/volumes"
Jul 28 20:49:06.837264 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:06.837251    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="724cbffa-8c0a-41f0-9173-8499723f819e" path="/var/lib/kubelet/pods/724cbffa-8c0a-41f0-9173-8499723f819e/volumes"
Jul 28 20:49:12.652990 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:12.652955    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:12.765911 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:12.765878    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:12.958558 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:12.958484    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:13.162886 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:13.162851    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:23.233584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.233549    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"]
Jul 28 20:49:23.234081 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.233793    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7" podUID="e84739c8-1edf-4a5f-92af-d5266a8d9b92" containerName="maas-api" containerID="cri-o://803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1" gracePeriod=30
Jul 28 20:49:23.270742 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.270689    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:23.347366 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.347329    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"]
Jul 28 20:49:23.347659 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.347626    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj" podUID="bd8a6a62-b358-40d8-9bd2-31465bb8f3ab" containerName="payload-pre-processing" containerID="cri-o://7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0" gracePeriod=30
Jul 28 20:49:23.383886 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.383852    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:23.472269 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.472205    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:23.509198 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.509174    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:49:23.558688 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.558662    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:23.611050 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.611029    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:49:23.654914 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.654890    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e84739c8-1edf-4a5f-92af-d5266a8d9b92-openshift-service-ca\") pod \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") "
Jul 28 20:49:23.654914 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.654921    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh6cr\" (UniqueName: \"kubernetes.io/projected/e84739c8-1edf-4a5f-92af-d5266a8d9b92-kube-api-access-nh6cr\") pod \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") "
Jul 28 20:49:23.655117 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.654941    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e84739c8-1edf-4a5f-92af-d5266a8d9b92-maas-api-tls\") pod \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\" (UID: \"e84739c8-1edf-4a5f-92af-d5266a8d9b92\") "
Jul 28 20:49:23.655271 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.655246    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84739c8-1edf-4a5f-92af-d5266a8d9b92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e84739c8-1edf-4a5f-92af-d5266a8d9b92" (UID: "e84739c8-1edf-4a5f-92af-d5266a8d9b92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:23.656974 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.656951    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84739c8-1edf-4a5f-92af-d5266a8d9b92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e84739c8-1edf-4a5f-92af-d5266a8d9b92" (UID: "e84739c8-1edf-4a5f-92af-d5266a8d9b92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:49:23.657083 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.657064    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84739c8-1edf-4a5f-92af-d5266a8d9b92-kube-api-access-nh6cr" (OuterVolumeSpecName: "kube-api-access-nh6cr") pod "e84739c8-1edf-4a5f-92af-d5266a8d9b92" (UID: "e84739c8-1edf-4a5f-92af-d5266a8d9b92"). InnerVolumeSpecName "kube-api-access-nh6cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:49:23.756276 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.756206    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-plugins-config-volume\") pod \"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab\" (UID: \"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab\") "
Jul 28 20:49:23.756410 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.756308    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhc98\" (UniqueName: \"kubernetes.io/projected/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-kube-api-access-nhc98\") pod \"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab\" (UID: \"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab\") "
Jul 28 20:49:23.756553 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.756532    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd8a6a62-b358-40d8-9bd2-31465bb8f3ab" (UID: "bd8a6a62-b358-40d8-9bd2-31465bb8f3ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:23.756627 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.756608    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e84739c8-1edf-4a5f-92af-d5266a8d9b92-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:49:23.756666 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.756634    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh6cr\" (UniqueName: \"kubernetes.io/projected/e84739c8-1edf-4a5f-92af-d5266a8d9b92-kube-api-access-nh6cr\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:49:23.756666 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.756648    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e84739c8-1edf-4a5f-92af-d5266a8d9b92-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:49:23.756666 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.756660    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:49:23.758264 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.758245    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-kube-api-access-nhc98" (OuterVolumeSpecName: "kube-api-access-nhc98") pod "bd8a6a62-b358-40d8-9bd2-31465bb8f3ab" (UID: "bd8a6a62-b358-40d8-9bd2-31465bb8f3ab"). InnerVolumeSpecName "kube-api-access-nhc98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:49:23.857495 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.857465    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhc98\" (UniqueName: \"kubernetes.io/projected/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab-kube-api-access-nhc98\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:49:23.946016 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:23.945950    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:24.271147 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.271121    2575 generic.go:356] "Generic (PLEG): container finished" podID="e84739c8-1edf-4a5f-92af-d5266a8d9b92" containerID="803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1" exitCode=0
Jul 28 20:49:24.271580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.271166    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"
Jul 28 20:49:24.271580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.271214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7" event={"ID":"e84739c8-1edf-4a5f-92af-d5266a8d9b92","Type":"ContainerDied","Data":"803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1"}
Jul 28 20:49:24.271580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.271288    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7" event={"ID":"e84739c8-1edf-4a5f-92af-d5266a8d9b92","Type":"ContainerDied","Data":"772f6d7e70978e28930c1b2b849bbae23ea7383df20f6d255545d9c42858bca8"}
Jul 28 20:49:24.271580 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.271319    2575 scope.go:117] "RemoveContainer" containerID="803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1"
Jul 28 20:49:24.272631 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.272531    2575 generic.go:356] "Generic (PLEG): container finished" podID="bd8a6a62-b358-40d8-9bd2-31465bb8f3ab" containerID="7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0" exitCode=0
Jul 28 20:49:24.272631 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.272564    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"
Jul 28 20:49:24.272631 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.272599    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj" event={"ID":"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab","Type":"ContainerDied","Data":"7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0"}
Jul 28 20:49:24.272804 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.272636    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj" event={"ID":"bd8a6a62-b358-40d8-9bd2-31465bb8f3ab","Type":"ContainerDied","Data":"5f1ed1e3eaa4523124ba6c36638cebf993b950497b671a392e8e8ae4d3720164"}
Jul 28 20:49:24.281275 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.281253    2575 scope.go:117] "RemoveContainer" containerID="803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1"
Jul 28 20:49:24.281558 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:49:24.281529    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1\": container with ID starting with 803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1 not found: ID does not exist" containerID="803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1"
Jul 28 20:49:24.281643 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.281568    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1"} err="failed to get container status \"803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1\": rpc error: code = NotFound desc = could not find container \"803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1\": container with ID starting with 803d985213de06e858c5d0eedd050c2fd1c9977adc6e1e19ff338906d3fe15c1 not found: ID does not exist"
Jul 28 20:49:24.281643 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.281585    2575 scope.go:117] "RemoveContainer" containerID="7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0"
Jul 28 20:49:24.289520 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.289500    2575 scope.go:117] "RemoveContainer" containerID="7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0"
Jul 28 20:49:24.289769 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:49:24.289747    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0\": container with ID starting with 7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0 not found: ID does not exist" containerID="7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0"
Jul 28 20:49:24.289852 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.289774    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0"} err="failed to get container status \"7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0\": rpc error: code = NotFound desc = could not find container \"7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0\": container with ID starting with 7b823e54ad6c29a8d5ba135f57102e1ca5e1dbd9efd0f2708d7ab8dfb456eaa0 not found: ID does not exist"
Jul 28 20:49:24.301237 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.301209    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"]
Jul 28 20:49:24.308610 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.308593    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-42ff09-76c9bcbd58-87tw7"]
Jul 28 20:49:24.324518 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.324498    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"]
Jul 28 20:49:24.330784 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.330761    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-42ff09-7b9485c96-5r4xj"]
Jul 28 20:49:24.836837 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.836807    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8a6a62-b358-40d8-9bd2-31465bb8f3ab" path="/var/lib/kubelet/pods/bd8a6a62-b358-40d8-9bd2-31465bb8f3ab/volumes"
Jul 28 20:49:24.837213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:24.837200    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e84739c8-1edf-4a5f-92af-d5266a8d9b92" path="/var/lib/kubelet/pods/e84739c8-1edf-4a5f-92af-d5266a8d9b92/volumes"
Jul 28 20:49:30.429153 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:30.429120    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:30.505762 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:30.505732    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:30.834493 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:30.834398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:31.069173 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:31.069143    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:31.349191 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:31.349142    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:31.766453 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:31.766419    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:32.228441 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:32.228404    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:34.955587 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:34.955552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:35.009341 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009309    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"]
Jul 28 20:49:35.009723 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009708    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e84739c8-1edf-4a5f-92af-d5266a8d9b92" containerName="maas-api"
Jul 28 20:49:35.009800 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009724    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e84739c8-1edf-4a5f-92af-d5266a8d9b92" containerName="maas-api"
Jul 28 20:49:35.009800 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009736    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8a6a62-b358-40d8-9bd2-31465bb8f3ab" containerName="payload-pre-processing"
Jul 28 20:49:35.009800 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009742    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8a6a62-b358-40d8-9bd2-31465bb8f3ab" containerName="payload-pre-processing"
Jul 28 20:49:35.009800 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009755    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f6b6c4-57c8-4938-892f-7ba0fa3ac87d" containerName="maas-api"
Jul 28 20:49:35.009800 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009763    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f6b6c4-57c8-4938-892f-7ba0fa3ac87d" containerName="maas-api"
Jul 28 20:49:35.009800 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009778    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="724cbffa-8c0a-41f0-9173-8499723f819e" containerName="payload-pre-processing"
Jul 28 20:49:35.009800 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009783    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="724cbffa-8c0a-41f0-9173-8499723f819e" containerName="payload-pre-processing"
Jul 28 20:49:35.010168 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009848    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8a6a62-b358-40d8-9bd2-31465bb8f3ab" containerName="payload-pre-processing"
Jul 28 20:49:35.010168 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009862    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f6b6c4-57c8-4938-892f-7ba0fa3ac87d" containerName="maas-api"
Jul 28 20:49:35.010168 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009868    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="724cbffa-8c0a-41f0-9173-8499723f819e" containerName="payload-pre-processing"
Jul 28 20:49:35.010168 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.009874    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e84739c8-1edf-4a5f-92af-d5266a8d9b92" containerName="maas-api"
Jul 28 20:49:35.013056 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.013034    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.017955 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.017930    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-cc370c-serving-cert\""
Jul 28 20:49:35.027214 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.027182    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"]
Jul 28 20:49:35.051676 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.051640    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:35.133849 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.133810    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:35.154460 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.154429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1d6a7f-2f16-4cbe-add6-375b70062942-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.154623 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.154482    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c1d6a7f-2f16-4cbe-add6-375b70062942-maas-api-tls\") pod \"maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.154623 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.154509    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbs5\" (UniqueName: \"kubernetes.io/projected/5c1d6a7f-2f16-4cbe-add6-375b70062942-kube-api-access-drbs5\") pod \"maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.255974 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.255879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1d6a7f-2f16-4cbe-add6-375b70062942-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.255974 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.255934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c1d6a7f-2f16-4cbe-add6-375b70062942-maas-api-tls\") pod \"maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.256204 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:49:35.256024    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-serving-cert: secret "maas-api-e2e-shared-a-cc370c-serving-cert" not found
Jul 28 20:49:35.256204 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.256066    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbs5\" (UniqueName: \"kubernetes.io/projected/5c1d6a7f-2f16-4cbe-add6-375b70062942-kube-api-access-drbs5\") pod \"maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.256204 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:49:35.256080    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c1d6a7f-2f16-4cbe-add6-375b70062942-maas-api-tls podName:5c1d6a7f-2f16-4cbe-add6-375b70062942 nodeName:}" failed. No retries permitted until 2026-07-28 20:49:35.756064031 +0000 UTC m=+2475.370301143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c1d6a7f-2f16-4cbe-add6-375b70062942-maas-api-tls") pod "maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s" (UID: "5c1d6a7f-2f16-4cbe-add6-375b70062942") : secret "maas-api-e2e-shared-a-cc370c-serving-cert" not found
Jul 28 20:49:35.256719 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.256697    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1d6a7f-2f16-4cbe-add6-375b70062942-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.267931 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.267906    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbs5\" (UniqueName: \"kubernetes.io/projected/5c1d6a7f-2f16-4cbe-add6-375b70062942-kube-api-access-drbs5\") pod \"maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.413291 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.413217    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:35.710042 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.710006    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:35.761164 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.761127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c1d6a7f-2f16-4cbe-add6-375b70062942-maas-api-tls\") pod \"maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.763539 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.763508    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c1d6a7f-2f16-4cbe-add6-375b70062942-maas-api-tls\") pod \"maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:35.801946 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.801913    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:35.900383 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.900351    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"]
Jul 28 20:49:35.903937 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.903920    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 20:49:35.906623 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.906591    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-cc370c-dockercfg-f5wc6\""
Jul 28 20:49:35.906737 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.906643    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-cc370c\""
Jul 28 20:49:35.914289 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.914268    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"]
Jul 28 20:49:35.927681 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:35.927655    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:36.069030 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:36.068999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbfd028-8b80-472d-a326-2a5cbe95ee57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x\" (UID: \"ddbfd028-8b80-472d-a326-2a5cbe95ee57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 20:49:36.069548 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:36.069117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqjl\" (UniqueName: \"kubernetes.io/projected/ddbfd028-8b80-472d-a326-2a5cbe95ee57-kube-api-access-fcqjl\") pod \"payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x\" (UID: \"ddbfd028-8b80-472d-a326-2a5cbe95ee57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 20:49:36.090554 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:36.090528    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"]
Jul 28 20:49:36.093261 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:49:36.093211    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1d6a7f_2f16_4cbe_add6_375b70062942.slice/crio-3240592f7c4c9b4b2e4fea691a7a3f59fc89d69365332091840ecdac115c2d57 WatchSource:0}: Error finding container 3240592f7c4c9b4b2e4fea691a7a3f59fc89d69365332091840ecdac115c2d57: Status 404 returned error can't find the container with id 3240592f7c4c9b4b2e4fea691a7a3f59fc89d69365332091840ecdac115c2d57
Jul 28 20:49:36.170515 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:36.170485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqjl\" (UniqueName: \"kubernetes.io/projected/ddbfd028-8b80-472d-a326-2a5cbe95ee57-kube-api-access-fcqjl\") pod \"payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x\" (UID: \"ddbfd028-8b80-472d-a326-2a5cbe95ee57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 20:49:36.170638 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:36.170559    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbfd028-8b80-472d-a326-2a5cbe95ee57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x\" (UID: \"ddbfd028-8b80-472d-a326-2a5cbe95ee57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 20:49:36.171265 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:36.171245    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbfd028-8b80-472d-a326-2a5cbe95ee57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x\" (UID: \"ddbfd028-8b80-472d-a326-2a5cbe95ee57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 20:49:36.179496 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:36.179477    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqjl\" (UniqueName: \"kubernetes.io/projected/ddbfd028-8b80-472d-a326-2a5cbe95ee57-kube-api-access-fcqjl\") pod \"payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x\" (UID: \"ddbfd028-8b80-472d-a326-2a5cbe95ee57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 20:49:36.215533 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:36.215504    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 20:49:36.335629 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:36.335569    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s" event={"ID":"5c1d6a7f-2f16-4cbe-add6-375b70062942","Type":"ContainerStarted","Data":"3240592f7c4c9b4b2e4fea691a7a3f59fc89d69365332091840ecdac115c2d57"}
Jul 28 20:49:36.355173 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:36.355142    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"]
Jul 28 20:49:36.357905 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:49:36.357883    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbfd028_8b80_472d_a326_2a5cbe95ee57.slice/crio-1ad314f1677dfa3e27c0c2b79f5d360ffb9960643944c499ed6055642985b82a WatchSource:0}: Error finding container 1ad314f1677dfa3e27c0c2b79f5d360ffb9960643944c499ed6055642985b82a: Status 404 returned error can't find the container with id 1ad314f1677dfa3e27c0c2b79f5d360ffb9960643944c499ed6055642985b82a
Jul 28 20:49:37.343489 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:37.343454    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x" event={"ID":"ddbfd028-8b80-472d-a326-2a5cbe95ee57","Type":"ContainerStarted","Data":"9e08887ebb672e258c82f2543183955188ea02c72523e9840ae78364fc6ad38b"}
Jul 28 20:49:37.343949 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:37.343525    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 20:49:37.343949 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:37.343568    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x" event={"ID":"ddbfd028-8b80-472d-a326-2a5cbe95ee57","Type":"ContainerStarted","Data":"1ad314f1677dfa3e27c0c2b79f5d360ffb9960643944c499ed6055642985b82a"}
Jul 28 20:49:37.345206 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:37.345174    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s" event={"ID":"5c1d6a7f-2f16-4cbe-add6-375b70062942","Type":"ContainerStarted","Data":"162007bef538619d0c4e7370dd383eb45def8ad0c346e1c40b655c602e2d2c1f"}
Jul 28 20:49:37.345344 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:37.345271    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:37.361041 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:37.361006    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x" podStartSLOduration=2.360992519 podStartE2EDuration="2.360992519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:37.360169797 +0000 UTC m=+2476.974406930" watchObservedRunningTime="2026-07-28 20:49:37.360992519 +0000 UTC m=+2476.975229652"
Jul 28 20:49:37.387822 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:37.387777    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s" podStartSLOduration=2.994114127 podStartE2EDuration="3.387764767s" totalImagesPullingTime="393.65064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:37.386077093 +0000 UTC m=+2477.000314227" watchObservedRunningTime="2026-07-28 20:49:37.387764767 +0000 UTC m=+2477.002001936"
Jul 28 20:49:43.355958 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:43.355929    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 20:49:48.352885 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:48.352847    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 20:49:54.611472 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:54.611440    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:54.678573 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:54.678545    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:49:59.918839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:49:59.918806    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:00.277897 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:00.277796    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:00.616993 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:00.616902    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:00.897970 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:00.897919    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:04.093394 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.093359    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:04.306727 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.306696    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"]
Jul 28 20:50:04.310147 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.310128    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.311694 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.311662    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:04.317142 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.317119    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-35c841-serving-cert\""
Jul 28 20:50:04.336926 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.336899    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"]
Jul 28 20:50:04.415743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.415708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c14450-d207-40d6-8e7c-4bb0140145c7-maas-api-tls\") pod \"maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.415743 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.415745    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm6fh\" (UniqueName: \"kubernetes.io/projected/08c14450-d207-40d6-8e7c-4bb0140145c7-kube-api-access-vm6fh\") pod \"maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.416007 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.415831    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c14450-d207-40d6-8e7c-4bb0140145c7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.502951 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.502909    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:04.516814 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.516787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c14450-d207-40d6-8e7c-4bb0140145c7-maas-api-tls\") pod \"maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.516990 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.516818    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm6fh\" (UniqueName: \"kubernetes.io/projected/08c14450-d207-40d6-8e7c-4bb0140145c7-kube-api-access-vm6fh\") pod \"maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.516990 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.516876    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c14450-d207-40d6-8e7c-4bb0140145c7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.517608 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.517589    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c14450-d207-40d6-8e7c-4bb0140145c7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.519357 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.519331    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c14450-d207-40d6-8e7c-4bb0140145c7-maas-api-tls\") pod \"maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.526958 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.526935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm6fh\" (UniqueName: \"kubernetes.io/projected/08c14450-d207-40d6-8e7c-4bb0140145c7-kube-api-access-vm6fh\") pod \"maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.596809 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.596777    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:04.622243 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.622202    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:04.758423 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:04.758399    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"]
Jul 28 20:50:04.761006 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:50:04.760977    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c14450_d207_40d6_8e7c_4bb0140145c7.slice/crio-59fcaeb46d55293cca5bd36b25cb91959672ef32598609a15695b2a92ff34992 WatchSource:0}: Error finding container 59fcaeb46d55293cca5bd36b25cb91959672ef32598609a15695b2a92ff34992: Status 404 returned error can't find the container with id 59fcaeb46d55293cca5bd36b25cb91959672ef32598609a15695b2a92ff34992
Jul 28 20:50:05.001715 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.001632    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:05.102152 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.102121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:05.109545 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.109518    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"]
Jul 28 20:50:05.114090 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.114074    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 20:50:05.116420 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.116401    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-35c841\""
Jul 28 20:50:05.129041 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.129019    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"]
Jul 28 20:50:05.223057 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.223020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcq9\" (UniqueName: \"kubernetes.io/projected/8b479785-71d5-4bd2-9445-4780b792ca49-kube-api-access-9kcq9\") pod \"payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j\" (UID: \"8b479785-71d5-4bd2-9445-4780b792ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 20:50:05.223253 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.223086    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b479785-71d5-4bd2-9445-4780b792ca49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j\" (UID: \"8b479785-71d5-4bd2-9445-4780b792ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 20:50:05.324198 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.324118    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcq9\" (UniqueName: \"kubernetes.io/projected/8b479785-71d5-4bd2-9445-4780b792ca49-kube-api-access-9kcq9\") pod \"payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j\" (UID: \"8b479785-71d5-4bd2-9445-4780b792ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 20:50:05.324198 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.324166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b479785-71d5-4bd2-9445-4780b792ca49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j\" (UID: \"8b479785-71d5-4bd2-9445-4780b792ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 20:50:05.324873 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.324846    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b479785-71d5-4bd2-9445-4780b792ca49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j\" (UID: \"8b479785-71d5-4bd2-9445-4780b792ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 20:50:05.335535 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.335511    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcq9\" (UniqueName: \"kubernetes.io/projected/8b479785-71d5-4bd2-9445-4780b792ca49-kube-api-access-9kcq9\") pod \"payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j\" (UID: \"8b479785-71d5-4bd2-9445-4780b792ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 20:50:05.424914 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.424883    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 20:50:05.487294 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.487261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t" event={"ID":"08c14450-d207-40d6-8e7c-4bb0140145c7","Type":"ContainerStarted","Data":"59fcaeb46d55293cca5bd36b25cb91959672ef32598609a15695b2a92ff34992"}
Jul 28 20:50:05.562126 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:05.562101    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"]
Jul 28 20:50:05.564796 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:50:05.564767    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b479785_71d5_4bd2_9445_4780b792ca49.slice/crio-8ced87a3dd881dae936cdd8278a97c728736f8ae2c229d1010725ae3ce7dfb5a WatchSource:0}: Error finding container 8ced87a3dd881dae936cdd8278a97c728736f8ae2c229d1010725ae3ce7dfb5a: Status 404 returned error can't find the container with id 8ced87a3dd881dae936cdd8278a97c728736f8ae2c229d1010725ae3ce7dfb5a
Jul 28 20:50:06.493451 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:06.493408    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j" event={"ID":"8b479785-71d5-4bd2-9445-4780b792ca49","Type":"ContainerStarted","Data":"39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2"}
Jul 28 20:50:06.493891 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:06.493473    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j" event={"ID":"8b479785-71d5-4bd2-9445-4780b792ca49","Type":"ContainerStarted","Data":"8ced87a3dd881dae936cdd8278a97c728736f8ae2c229d1010725ae3ce7dfb5a"}
Jul 28 20:50:06.493891 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:06.493502    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 20:50:06.495010 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:06.494976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t" event={"ID":"08c14450-d207-40d6-8e7c-4bb0140145c7","Type":"ContainerStarted","Data":"6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6"}
Jul 28 20:50:06.495133 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:06.495037    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:06.512442 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:06.512399    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j" podStartSLOduration=1.512384504 podStartE2EDuration="1.512384504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:50:06.511742465 +0000 UTC m=+2506.125979597" watchObservedRunningTime="2026-07-28 20:50:06.512384504 +0000 UTC m=+2506.126621636"
Jul 28 20:50:06.531557 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:06.531509    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t" podStartSLOduration=1.57793875 podStartE2EDuration="2.531491375s" totalImagesPullingTime="953.552625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:50:06.528973859 +0000 UTC m=+2506.143210993" watchObservedRunningTime="2026-07-28 20:50:06.531491375 +0000 UTC m=+2506.145728509"
Jul 28 20:50:10.244998 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:10.244952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:12.505662 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:12.505589    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 20:50:17.502614 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:17.502578    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 20:50:28.011182 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:28.011147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:28.112470 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:28.112440    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:29.077098 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:29.077063    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:39.030050 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:39.030012    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:57.315321 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:57.315282    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:50:58.846604 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:50:58.846577    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:08.518352 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:08.518322    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:08.618114 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:08.618086    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:10.601416 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.601379    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"]
Jul 28 20:51:10.605047 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.605026    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.608670 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.608645    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"kube-root-ca.crt\""
Jul 28 20:51:10.608670 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.608660    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"default-dockercfg-4d694\""
Jul 28 20:51:10.608859 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.608658    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"auth-test-model-35c841-kserve-self-signed-certs\""
Jul 28 20:51:10.608859 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.608759    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"openshift-service-ca.crt\""
Jul 28 20:51:10.614584 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.614561    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"]
Jul 28 20:51:10.635132 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.635102    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:10.702711 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.702677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-home\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.702862 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.702724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-dshm\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.702862 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.702752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-model-cache\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.702862 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.702812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5161a99-909b-4482-b238-9fd2f6c5b028-tls-certs\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.702862 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.702853    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-tmp-dir\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.703032 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.702893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgbq\" (UniqueName: \"kubernetes.io/projected/d5161a99-909b-4482-b238-9fd2f6c5b028-kube-api-access-7vgbq\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.803581 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.803541    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgbq\" (UniqueName: \"kubernetes.io/projected/d5161a99-909b-4482-b238-9fd2f6c5b028-kube-api-access-7vgbq\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.803781 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.803620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-home\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.803781 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.803656    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-dshm\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.803781 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.803681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-model-cache\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.803781 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.803712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5161a99-909b-4482-b238-9fd2f6c5b028-tls-certs\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.803781 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.803748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-tmp-dir\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.804090 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.804069    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-home\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.804141 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.804081    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-model-cache\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.804372 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.804355    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-tmp-dir\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.806013 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.805993    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-dshm\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.806497 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.806477    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5161a99-909b-4482-b238-9fd2f6c5b028-tls-certs\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.812445 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.812426    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgbq\" (UniqueName: \"kubernetes.io/projected/d5161a99-909b-4482-b238-9fd2f6c5b028-kube-api-access-7vgbq\") pod \"auth-test-model-35c841-kserve-589f474667-nnrvv\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") " pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:10.915147 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:10.915114    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:11.053631 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:11.053606    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"]
Jul 28 20:51:11.055848 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:51:11.055817    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5161a99_909b_4482_b238_9fd2f6c5b028.slice/crio-c57038049fdde69385829079ca3b461cf481002f99a334627b407b128f1452e9 WatchSource:0}: Error finding container c57038049fdde69385829079ca3b461cf481002f99a334627b407b128f1452e9: Status 404 returned error can't find the container with id c57038049fdde69385829079ca3b461cf481002f99a334627b407b128f1452e9
Jul 28 20:51:11.817581 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:11.817542    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv" event={"ID":"d5161a99-909b-4482-b238-9fd2f6c5b028","Type":"ContainerStarted","Data":"abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b"}
Jul 28 20:51:11.818046 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:11.817586    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv" event={"ID":"d5161a99-909b-4482-b238-9fd2f6c5b028","Type":"ContainerStarted","Data":"c57038049fdde69385829079ca3b461cf481002f99a334627b407b128f1452e9"}
Jul 28 20:51:11.859173 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:11.859123    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv" podStartSLOduration=1.859106779 podStartE2EDuration="1.859106779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:51:11.858653292 +0000 UTC m=+2571.472890425" watchObservedRunningTime="2026-07-28 20:51:11.859106779 +0000 UTC m=+2571.473343913"
Jul 28 20:51:17.501630 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:17.501599    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:17.682735 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:17.682697    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:51:20.915722 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:20.915687    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:20.915722 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:20.915731    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:20.928496 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:20.928473    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:51:21.136884 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:21.136849    2575 scope.go:117] "RemoveContainer" containerID="d31966d4294ab808f0d2c81eee08d04c60cf727195c5ffffdea977548e1bf4b5"
Jul 28 20:51:21.876003 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:51:21.875974    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:52:46.420254 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:52:46.420188    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:45.470019 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:55:45.469975    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:55:46.374136 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:55:46.374105    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"]
Jul 28 20:55:46.400206 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:55:46.400176    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:01.393902 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.393845    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv" podUID="d5161a99-909b-4482-b238-9fd2f6c5b028" containerName="main" containerID="cri-o://abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b" gracePeriod=45
Jul 28 20:56:01.662379 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.662350    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:56:01.730649 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.730612    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-model-cache\") pod \"d5161a99-909b-4482-b238-9fd2f6c5b028\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") "
Jul 28 20:56:01.730854 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.730667    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-home\") pod \"d5161a99-909b-4482-b238-9fd2f6c5b028\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") "
Jul 28 20:56:01.730854 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.730693    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5161a99-909b-4482-b238-9fd2f6c5b028-tls-certs\") pod \"d5161a99-909b-4482-b238-9fd2f6c5b028\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") "
Jul 28 20:56:01.730854 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.730746    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-dshm\") pod \"d5161a99-909b-4482-b238-9fd2f6c5b028\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") "
Jul 28 20:56:01.730854 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.730784    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vgbq\" (UniqueName: \"kubernetes.io/projected/d5161a99-909b-4482-b238-9fd2f6c5b028-kube-api-access-7vgbq\") pod \"d5161a99-909b-4482-b238-9fd2f6c5b028\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") "
Jul 28 20:56:01.730854 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.730826    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-tmp-dir\") pod \"d5161a99-909b-4482-b238-9fd2f6c5b028\" (UID: \"d5161a99-909b-4482-b238-9fd2f6c5b028\") "
Jul 28 20:56:01.731113 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.730926    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-home" (OuterVolumeSpecName: "home") pod "d5161a99-909b-4482-b238-9fd2f6c5b028" (UID: "d5161a99-909b-4482-b238-9fd2f6c5b028"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:56:01.731113 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.730938    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5161a99-909b-4482-b238-9fd2f6c5b028" (UID: "d5161a99-909b-4482-b238-9fd2f6c5b028"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:56:01.731222 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.731114    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-home\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:56:01.731222 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.731129    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-model-cache\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:56:01.731332 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.731241    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5161a99-909b-4482-b238-9fd2f6c5b028" (UID: "d5161a99-909b-4482-b238-9fd2f6c5b028"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:56:01.733213 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.733183    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5161a99-909b-4482-b238-9fd2f6c5b028-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5161a99-909b-4482-b238-9fd2f6c5b028" (UID: "d5161a99-909b-4482-b238-9fd2f6c5b028"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:56:01.733375 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.733326    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-dshm" (OuterVolumeSpecName: "dshm") pod "d5161a99-909b-4482-b238-9fd2f6c5b028" (UID: "d5161a99-909b-4482-b238-9fd2f6c5b028"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:56:01.733444 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.733420    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5161a99-909b-4482-b238-9fd2f6c5b028-kube-api-access-7vgbq" (OuterVolumeSpecName: "kube-api-access-7vgbq") pod "d5161a99-909b-4482-b238-9fd2f6c5b028" (UID: "d5161a99-909b-4482-b238-9fd2f6c5b028"). InnerVolumeSpecName "kube-api-access-7vgbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:56:01.831616 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.831581    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vgbq\" (UniqueName: \"kubernetes.io/projected/d5161a99-909b-4482-b238-9fd2f6c5b028-kube-api-access-7vgbq\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:56:01.831779 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.831637    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-tmp-dir\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:56:01.831779 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.831652    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5161a99-909b-4482-b238-9fd2f6c5b028-tls-certs\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:56:01.831779 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:01.831664    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5161a99-909b-4482-b238-9fd2f6c5b028-dshm\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:56:02.198037 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:02.198005    2575 generic.go:356] "Generic (PLEG): container finished" podID="d5161a99-909b-4482-b238-9fd2f6c5b028" containerID="abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b" exitCode=0
Jul 28 20:56:02.198211 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:02.198063    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"
Jul 28 20:56:02.198211 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:02.198087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv" event={"ID":"d5161a99-909b-4482-b238-9fd2f6c5b028","Type":"ContainerDied","Data":"abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b"}
Jul 28 20:56:02.198211 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:02.198133    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv" event={"ID":"d5161a99-909b-4482-b238-9fd2f6c5b028","Type":"ContainerDied","Data":"c57038049fdde69385829079ca3b461cf481002f99a334627b407b128f1452e9"}
Jul 28 20:56:02.198211 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:02.198151    2575 scope.go:117] "RemoveContainer" containerID="abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b"
Jul 28 20:56:02.207258 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:02.207212    2575 scope.go:117] "RemoveContainer" containerID="abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b"
Jul 28 20:56:02.207523 ip-10-0-140-243 kubenswrapper[2575]: E0728 20:56:02.207504    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b\": container with ID starting with abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b not found: ID does not exist" containerID="abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b"
Jul 28 20:56:02.207579 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:02.207532    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b"} err="failed to get container status \"abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b\": rpc error: code = NotFound desc = could not find container \"abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b\": container with ID starting with abf8770b5523d643080fb04eca2565222af8ff501a87b78ba76804fa415d814b not found: ID does not exist"
Jul 28 20:56:02.218788 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:02.218763    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"]
Jul 28 20:56:02.222977 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:02.222956    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/auth-test-model-35c841-kserve-589f474667-nnrvv"]
Jul 28 20:56:02.836205 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:02.836172    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5161a99-909b-4482-b238-9fd2f6c5b028" path="/var/lib/kubelet/pods/d5161a99-909b-4482-b238-9fd2f6c5b028/volumes"
Jul 28 20:56:03.505492 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:03.505463    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:32.093488 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.093456    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"]
Jul 28 20:56:32.093949 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.093892    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5161a99-909b-4482-b238-9fd2f6c5b028" containerName="main"
Jul 28 20:56:32.093949 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.093905    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5161a99-909b-4482-b238-9fd2f6c5b028" containerName="main"
Jul 28 20:56:32.094032 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.093970    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5161a99-909b-4482-b238-9fd2f6c5b028" containerName="main"
Jul 28 20:56:32.096926 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.096910    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.099803 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.099781    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"sub-test-model-35c841-kserve-self-signed-certs\""
Jul 28 20:56:32.099944 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.099781    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"openshift-service-ca.crt\""
Jul 28 20:56:32.099944 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.099783    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"kube-root-ca.crt\""
Jul 28 20:56:32.100071 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.100055    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"default-dockercfg-4d694\""
Jul 28 20:56:32.108378 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.108345    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"]
Jul 28 20:56:32.119389 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.119364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:56:32.198778 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.198734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-model-cache\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.198971 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.198806    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbj5\" (UniqueName: \"kubernetes.io/projected/58da5c7f-b8a4-4440-afa4-d7474fec4594-kube-api-access-6cbj5\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.198971 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.198842    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-home\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.198971 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.198900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-tmp-dir\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.198971 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.198915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58da5c7f-b8a4-4440-afa4-d7474fec4594-tls-certs\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.198971 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.198936    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-dshm\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.300163 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.300124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-home\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.300387 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.300174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-tmp-dir\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.300387 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.300190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58da5c7f-b8a4-4440-afa4-d7474fec4594-tls-certs\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.300387 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.300211    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-dshm\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.300387 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.300277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-model-cache\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.300619 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.300417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbj5\" (UniqueName: \"kubernetes.io/projected/58da5c7f-b8a4-4440-afa4-d7474fec4594-kube-api-access-6cbj5\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.300619 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.300589    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-tmp-dir\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.300717 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.300619    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-home\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.300717 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.300693    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-model-cache\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.302554 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.302532    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-dshm\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.302846 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.302827    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58da5c7f-b8a4-4440-afa4-d7474fec4594-tls-certs\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.309126 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.309102    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbj5\" (UniqueName: \"kubernetes.io/projected/58da5c7f-b8a4-4440-afa4-d7474fec4594-kube-api-access-6cbj5\") pod \"sub-test-model-35c841-kserve-797675674c-nbwpg\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") " pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.407793 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.407759    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:32.541325 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:32.541299    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"]
Jul 28 20:56:32.543541 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:56:32.543517    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58da5c7f_b8a4_4440_afa4_d7474fec4594.slice/crio-61f316367b6f634968bdc90c9b984bed9ac89c9bdeb0e4e2cc3f1a24d34a6da3 WatchSource:0}: Error finding container 61f316367b6f634968bdc90c9b984bed9ac89c9bdeb0e4e2cc3f1a24d34a6da3: Status 404 returned error can't find the container with id 61f316367b6f634968bdc90c9b984bed9ac89c9bdeb0e4e2cc3f1a24d34a6da3
Jul 28 20:56:33.351285 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:33.351251    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg" event={"ID":"58da5c7f-b8a4-4440-afa4-d7474fec4594","Type":"ContainerStarted","Data":"4aeb566b60556dbcb1bc1167aeab4fd824b71abb12e6e41f5dbcb76c8a8a01c7"}
Jul 28 20:56:33.351285 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:33.351294    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg" event={"ID":"58da5c7f-b8a4-4440-afa4-d7474fec4594","Type":"ContainerStarted","Data":"61f316367b6f634968bdc90c9b984bed9ac89c9bdeb0e4e2cc3f1a24d34a6da3"}
Jul 28 20:56:33.368323 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:33.368279    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg" podStartSLOduration=1.3682648450000001 podStartE2EDuration="1.368264845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:56:33.36600448 +0000 UTC m=+2892.980241613" watchObservedRunningTime="2026-07-28 20:56:33.368264845 +0000 UTC m=+2892.982501978"
Jul 28 20:56:42.408074 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:42.408041    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:42.408501 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:42.408086    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:42.420540 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:42.420516    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:56:43.408331 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:56:43.408302    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:58:06.560400 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:06.560321    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:07.636308 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:07.636267    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"]
Jul 28 20:58:07.664668 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:07.664620    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:18.803189 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:18.803148    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:58:22.654970 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.654927    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg" podUID="58da5c7f-b8a4-4440-afa4-d7474fec4594" containerName="main" containerID="cri-o://4aeb566b60556dbcb1bc1167aeab4fd824b71abb12e6e41f5dbcb76c8a8a01c7" gracePeriod=45
Jul 28 20:58:22.874618 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.874590    2575 generic.go:356] "Generic (PLEG): container finished" podID="58da5c7f-b8a4-4440-afa4-d7474fec4594" containerID="4aeb566b60556dbcb1bc1167aeab4fd824b71abb12e6e41f5dbcb76c8a8a01c7" exitCode=0
Jul 28 20:58:22.874783 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.874668    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg" event={"ID":"58da5c7f-b8a4-4440-afa4-d7474fec4594","Type":"ContainerDied","Data":"4aeb566b60556dbcb1bc1167aeab4fd824b71abb12e6e41f5dbcb76c8a8a01c7"}
Jul 28 20:58:22.902401 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.902375    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:58:22.948438 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.948365    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-dshm\") pod \"58da5c7f-b8a4-4440-afa4-d7474fec4594\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") "
Jul 28 20:58:22.948438 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.948404    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-home\") pod \"58da5c7f-b8a4-4440-afa4-d7474fec4594\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") "
Jul 28 20:58:22.948438 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.948424    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cbj5\" (UniqueName: \"kubernetes.io/projected/58da5c7f-b8a4-4440-afa4-d7474fec4594-kube-api-access-6cbj5\") pod \"58da5c7f-b8a4-4440-afa4-d7474fec4594\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") "
Jul 28 20:58:22.948713 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.948469    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58da5c7f-b8a4-4440-afa4-d7474fec4594-tls-certs\") pod \"58da5c7f-b8a4-4440-afa4-d7474fec4594\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") "
Jul 28 20:58:22.948713 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.948500    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-tmp-dir\") pod \"58da5c7f-b8a4-4440-afa4-d7474fec4594\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") "
Jul 28 20:58:22.948713 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.948534    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-model-cache\") pod \"58da5c7f-b8a4-4440-afa4-d7474fec4594\" (UID: \"58da5c7f-b8a4-4440-afa4-d7474fec4594\") "
Jul 28 20:58:22.948871 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.948690    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-home" (OuterVolumeSpecName: "home") pod "58da5c7f-b8a4-4440-afa4-d7474fec4594" (UID: "58da5c7f-b8a4-4440-afa4-d7474fec4594"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:58:22.948871 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.948780    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58da5c7f-b8a4-4440-afa4-d7474fec4594" (UID: "58da5c7f-b8a4-4440-afa4-d7474fec4594"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:58:22.948871 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.948805    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-home\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:58:22.949005 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.948903    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-model-cache" (OuterVolumeSpecName: "model-cache") pod "58da5c7f-b8a4-4440-afa4-d7474fec4594" (UID: "58da5c7f-b8a4-4440-afa4-d7474fec4594"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:58:22.950613 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.950586    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58da5c7f-b8a4-4440-afa4-d7474fec4594-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58da5c7f-b8a4-4440-afa4-d7474fec4594" (UID: "58da5c7f-b8a4-4440-afa4-d7474fec4594"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:58:22.950948 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.950927    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-dshm" (OuterVolumeSpecName: "dshm") pod "58da5c7f-b8a4-4440-afa4-d7474fec4594" (UID: "58da5c7f-b8a4-4440-afa4-d7474fec4594"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:58:22.951018 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:22.950930    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58da5c7f-b8a4-4440-afa4-d7474fec4594-kube-api-access-6cbj5" (OuterVolumeSpecName: "kube-api-access-6cbj5") pod "58da5c7f-b8a4-4440-afa4-d7474fec4594" (UID: "58da5c7f-b8a4-4440-afa4-d7474fec4594"). InnerVolumeSpecName "kube-api-access-6cbj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:58:23.050209 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:23.050176    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-dshm\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:58:23.050209 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:23.050211    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbj5\" (UniqueName: \"kubernetes.io/projected/58da5c7f-b8a4-4440-afa4-d7474fec4594-kube-api-access-6cbj5\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:58:23.050420 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:23.050245    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58da5c7f-b8a4-4440-afa4-d7474fec4594-tls-certs\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:58:23.050420 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:23.050254    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-tmp-dir\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:58:23.050420 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:23.050263    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58da5c7f-b8a4-4440-afa4-d7474fec4594-model-cache\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 20:58:23.880465 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:23.880428    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg" event={"ID":"58da5c7f-b8a4-4440-afa4-d7474fec4594","Type":"ContainerDied","Data":"61f316367b6f634968bdc90c9b984bed9ac89c9bdeb0e4e2cc3f1a24d34a6da3"}
Jul 28 20:58:23.880465 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:23.880454    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"
Jul 28 20:58:23.880933 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:23.880479    2575 scope.go:117] "RemoveContainer" containerID="4aeb566b60556dbcb1bc1167aeab4fd824b71abb12e6e41f5dbcb76c8a8a01c7"
Jul 28 20:58:23.902293 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:23.902270    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"]
Jul 28 20:58:23.905928 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:23.905897    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/sub-test-model-35c841-kserve-797675674c-nbwpg"]
Jul 28 20:58:24.836060 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:24.836029    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58da5c7f-b8a4-4440-afa4-d7474fec4594" path="/var/lib/kubelet/pods/58da5c7f-b8a4-4440-afa4-d7474fec4594/volumes"
Jul 28 20:58:42.649719 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:58:42.649685    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:59:11.489769 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.489732    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"]
Jul 28 20:59:11.490147 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.490137    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58da5c7f-b8a4-4440-afa4-d7474fec4594" containerName="main"
Jul 28 20:59:11.490147 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.490149    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="58da5c7f-b8a4-4440-afa4-d7474fec4594" containerName="main"
Jul 28 20:59:11.490241 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.490213    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="58da5c7f-b8a4-4440-afa4-d7474fec4594" containerName="main"
Jul 28 20:59:11.493381 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.493361    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.496535 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.496512    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"openshift-service-ca.crt\""
Jul 28 20:59:11.496663 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.496537    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"kube-root-ca.crt\""
Jul 28 20:59:11.496663 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.496541    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"rate-test-model-35c841-kserve-self-signed-certs\""
Jul 28 20:59:11.496663 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.496512    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35c841\"/\"default-dockercfg-4d694\""
Jul 28 20:59:11.504210 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.504188    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"]
Jul 28 20:59:11.515071 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.515034    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:59:11.652379 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.652346    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa37179-c7c0-4c7a-a98c-5648b1554236-tls-certs\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.652562 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.652388    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-model-cache\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.652562 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.652413    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-home\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.652562 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.652507    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-tmp-dir\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.652562 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.652542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-dshm\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.652745 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.652597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhq5\" (UniqueName: \"kubernetes.io/projected/9aa37179-c7c0-4c7a-a98c-5648b1554236-kube-api-access-vrhq5\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.753328 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.753255    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhq5\" (UniqueName: \"kubernetes.io/projected/9aa37179-c7c0-4c7a-a98c-5648b1554236-kube-api-access-vrhq5\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.753328 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.753317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa37179-c7c0-4c7a-a98c-5648b1554236-tls-certs\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.753547 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.753343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-model-cache\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.753547 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.753361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-home\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.753547 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.753391    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-tmp-dir\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.753547 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.753410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-dshm\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.753784 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.753763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-home\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.753839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.753809    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-tmp-dir\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.753839 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.753820    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-model-cache\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.755634 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.755609    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-dshm\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.755878 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.755861    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa37179-c7c0-4c7a-a98c-5648b1554236-tls-certs\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.760788 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.760770    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhq5\" (UniqueName: \"kubernetes.io/projected/9aa37179-c7c0-4c7a-a98c-5648b1554236-kube-api-access-vrhq5\") pod \"rate-test-model-35c841-kserve-7cff8ddc4c-f9twr\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") " pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.803759 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.803736    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:11.940363 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:11.940339    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"]
Jul 28 20:59:11.943374 ip-10-0-140-243 kubenswrapper[2575]: W0728 20:59:11.943347    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9aa37179_c7c0_4c7a_a98c_5648b1554236.slice/crio-4c84516209bc7ae2f5bba3bb74558f9e62bc48151289a8c7e97633e8ebca1898 WatchSource:0}: Error finding container 4c84516209bc7ae2f5bba3bb74558f9e62bc48151289a8c7e97633e8ebca1898: Status 404 returned error can't find the container with id 4c84516209bc7ae2f5bba3bb74558f9e62bc48151289a8c7e97633e8ebca1898
Jul 28 20:59:12.114001 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:12.113915    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr" event={"ID":"9aa37179-c7c0-4c7a-a98c-5648b1554236","Type":"ContainerStarted","Data":"806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f"}
Jul 28 20:59:12.114001 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:12.113964    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr" event={"ID":"9aa37179-c7c0-4c7a-a98c-5648b1554236","Type":"ContainerStarted","Data":"4c84516209bc7ae2f5bba3bb74558f9e62bc48151289a8c7e97633e8ebca1898"}
Jul 28 20:59:12.133029 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:12.132984    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr" podStartSLOduration=1.13296972 podStartE2EDuration="1.13296972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:59:12.131746988 +0000 UTC m=+3051.745984131" watchObservedRunningTime="2026-07-28 20:59:12.13296972 +0000 UTC m=+3051.747206852"
Jul 28 20:59:15.228007 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:15.227922    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 20:59:21.804402 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:21.804371    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:21.804794 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:21.804630    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:21.817123 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:21.817098    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 20:59:22.177577 ip-10-0-140-243 kubenswrapper[2575]: I0728 20:59:22.177548    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 21:00:12.025674 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:12.025643    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:00:12.125673 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:12.125614    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:00:39.606266 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:39.606209    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:00:40.586587 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:40.586553    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"]
Jul 28 21:00:40.635422 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:40.635384    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:00:55.605191 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:55.605124    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr" podUID="9aa37179-c7c0-4c7a-a98c-5648b1554236" containerName="main" containerID="cri-o://806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f" gracePeriod=45
Jul 28 21:00:55.880873 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:55.880845    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 21:00:56.068026 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.067992    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-home\") pod \"9aa37179-c7c0-4c7a-a98c-5648b1554236\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") "
Jul 28 21:00:56.068189 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068065    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-dshm\") pod \"9aa37179-c7c0-4c7a-a98c-5648b1554236\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") "
Jul 28 21:00:56.068189 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068086    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa37179-c7c0-4c7a-a98c-5648b1554236-tls-certs\") pod \"9aa37179-c7c0-4c7a-a98c-5648b1554236\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") "
Jul 28 21:00:56.068189 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068142    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-model-cache\") pod \"9aa37179-c7c0-4c7a-a98c-5648b1554236\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") "
Jul 28 21:00:56.068189 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068160    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-tmp-dir\") pod \"9aa37179-c7c0-4c7a-a98c-5648b1554236\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") "
Jul 28 21:00:56.068464 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068317    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrhq5\" (UniqueName: \"kubernetes.io/projected/9aa37179-c7c0-4c7a-a98c-5648b1554236-kube-api-access-vrhq5\") pod \"9aa37179-c7c0-4c7a-a98c-5648b1554236\" (UID: \"9aa37179-c7c0-4c7a-a98c-5648b1554236\") "
Jul 28 21:00:56.068464 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068361    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-home" (OuterVolumeSpecName: "home") pod "9aa37179-c7c0-4c7a-a98c-5648b1554236" (UID: "9aa37179-c7c0-4c7a-a98c-5648b1554236"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:00:56.068464 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068399    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-model-cache" (OuterVolumeSpecName: "model-cache") pod "9aa37179-c7c0-4c7a-a98c-5648b1554236" (UID: "9aa37179-c7c0-4c7a-a98c-5648b1554236"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:00:56.068464 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068429    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9aa37179-c7c0-4c7a-a98c-5648b1554236" (UID: "9aa37179-c7c0-4c7a-a98c-5648b1554236"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:00:56.068638 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068611    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-model-cache\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:00:56.068638 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068627    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-tmp-dir\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:00:56.068638 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.068634    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-home\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:00:56.070118 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.070091    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-dshm" (OuterVolumeSpecName: "dshm") pod "9aa37179-c7c0-4c7a-a98c-5648b1554236" (UID: "9aa37179-c7c0-4c7a-a98c-5648b1554236"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:00:56.070254 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.070171    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa37179-c7c0-4c7a-a98c-5648b1554236-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9aa37179-c7c0-4c7a-a98c-5648b1554236" (UID: "9aa37179-c7c0-4c7a-a98c-5648b1554236"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:00:56.070320 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.070298    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa37179-c7c0-4c7a-a98c-5648b1554236-kube-api-access-vrhq5" (OuterVolumeSpecName: "kube-api-access-vrhq5") pod "9aa37179-c7c0-4c7a-a98c-5648b1554236" (UID: "9aa37179-c7c0-4c7a-a98c-5648b1554236"). InnerVolumeSpecName "kube-api-access-vrhq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:00:56.169476 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.169440    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aa37179-c7c0-4c7a-a98c-5648b1554236-dshm\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:00:56.169476 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.169469    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa37179-c7c0-4c7a-a98c-5648b1554236-tls-certs\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:00:56.169476 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.169481    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrhq5\" (UniqueName: \"kubernetes.io/projected/9aa37179-c7c0-4c7a-a98c-5648b1554236-kube-api-access-vrhq5\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:00:56.624061 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.623969    2575 generic.go:356] "Generic (PLEG): container finished" podID="9aa37179-c7c0-4c7a-a98c-5648b1554236" containerID="806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f" exitCode=0
Jul 28 21:00:56.624061 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.624011    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"
Jul 28 21:00:56.624563 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.624059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr" event={"ID":"9aa37179-c7c0-4c7a-a98c-5648b1554236","Type":"ContainerDied","Data":"806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f"}
Jul 28 21:00:56.624563 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.624099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr" event={"ID":"9aa37179-c7c0-4c7a-a98c-5648b1554236","Type":"ContainerDied","Data":"4c84516209bc7ae2f5bba3bb74558f9e62bc48151289a8c7e97633e8ebca1898"}
Jul 28 21:00:56.624563 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.624118    2575 scope.go:117] "RemoveContainer" containerID="806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f"
Jul 28 21:00:56.633328 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.633295    2575 scope.go:117] "RemoveContainer" containerID="806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f"
Jul 28 21:00:56.633562 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:00:56.633544    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f\": container with ID starting with 806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f not found: ID does not exist" containerID="806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f"
Jul 28 21:00:56.633629 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.633573    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f"} err="failed to get container status \"806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f\": rpc error: code = NotFound desc = could not find container \"806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f\": container with ID starting with 806d97eb45f3a11513c492db50b6d50d5816a34bfd5b852c57e23b2263c23b8f not found: ID does not exist"
Jul 28 21:00:56.651218 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.651193    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"]
Jul 28 21:00:56.652799 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.652779    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-35c841/rate-test-model-35c841-kserve-7cff8ddc4c-f9twr"]
Jul 28 21:00:56.836002 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:00:56.835960    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa37179-c7c0-4c7a-a98c-5648b1554236" path="/var/lib/kubelet/pods/9aa37179-c7c0-4c7a-a98c-5648b1554236/volumes"
Jul 28 21:01:02.042932 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:02.042897    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:02.455859 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:02.455830    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:03.140128 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:03.140093    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:05.664005 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.663954    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:05.734589 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.734552    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"]
Jul 28 21:01:05.734965 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.734950    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa37179-c7c0-4c7a-a98c-5648b1554236" containerName="main"
Jul 28 21:01:05.734965 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.734966    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa37179-c7c0-4c7a-a98c-5648b1554236" containerName="main"
Jul 28 21:01:05.735096 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.735037    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa37179-c7c0-4c7a-a98c-5648b1554236" containerName="main"
Jul 28 21:01:05.738133 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.738116    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:05.740438 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.740415    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ce337a-serving-cert\""
Jul 28 21:01:05.755271 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.755247    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"]
Jul 28 21:01:05.838243 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.838184    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:05.838448 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.838310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:05.838448 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.838357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k467q\" (UniqueName: \"kubernetes.io/projected/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-kube-api-access-k467q\") pod \"maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:05.939047 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.938967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:05.939047 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.939015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:05.939047 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.939042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k467q\" (UniqueName: \"kubernetes.io/projected/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-kube-api-access-k467q\") pod \"maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:05.939583 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.939564    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:05.941456 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.941438    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:05.948349 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:05.948329    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k467q\" (UniqueName: \"kubernetes.io/projected/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-kube-api-access-k467q\") pod \"maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:06.049021 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.048992    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:06.185271 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.185245    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"]
Jul 28 21:01:06.187367 ip-10-0-140-243 kubenswrapper[2575]: W0728 21:01:06.187339    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9a8d93_4f56_46bd_ac4f_eef5fdd90289.slice/crio-308a2c1cb4796b14fe61e5c4a5420639ecc36a5db1545862be43b0d590d6a4d8 WatchSource:0}: Error finding container 308a2c1cb4796b14fe61e5c4a5420639ecc36a5db1545862be43b0d590d6a4d8: Status 404 returned error can't find the container with id 308a2c1cb4796b14fe61e5c4a5420639ecc36a5db1545862be43b0d590d6a4d8
Jul 28 21:01:06.188714 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.188694    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 21:01:06.455622 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.455544    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"]
Jul 28 21:01:06.460109 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.460089    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:01:06.462361 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.462338    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ce337a\""
Jul 28 21:01:06.462613 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.462597    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ce337a-dockercfg-s29kc\""
Jul 28 21:01:06.470523 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.470498    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"]
Jul 28 21:01:06.490371 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.490338    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"]
Jul 28 21:01:06.493858 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.493837    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:01:06.505439 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.505415    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"]
Jul 28 21:01:06.545072 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.545036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af2bce9-d428-49fb-9b4e-c847785c9541-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch\" (UID: \"2af2bce9-d428-49fb-9b4e-c847785c9541\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:01:06.545216 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.545080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6srx\" (UniqueName: \"kubernetes.io/projected/d49b11ba-4f16-4a32-b515-74c95848d2a7-kube-api-access-h6srx\") pod \"payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6\" (UID: \"d49b11ba-4f16-4a32-b515-74c95848d2a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:01:06.545216 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.545128    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2bb9\" (UniqueName: \"kubernetes.io/projected/2af2bce9-d428-49fb-9b4e-c847785c9541-kube-api-access-l2bb9\") pod \"payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch\" (UID: \"2af2bce9-d428-49fb-9b4e-c847785c9541\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:01:06.545332 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.545255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d49b11ba-4f16-4a32-b515-74c95848d2a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6\" (UID: \"d49b11ba-4f16-4a32-b515-74c95848d2a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:01:06.646539 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.646509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d49b11ba-4f16-4a32-b515-74c95848d2a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6\" (UID: \"d49b11ba-4f16-4a32-b515-74c95848d2a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:01:06.646734 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.646546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af2bce9-d428-49fb-9b4e-c847785c9541-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch\" (UID: \"2af2bce9-d428-49fb-9b4e-c847785c9541\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:01:06.646734 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.646578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6srx\" (UniqueName: \"kubernetes.io/projected/d49b11ba-4f16-4a32-b515-74c95848d2a7-kube-api-access-h6srx\") pod \"payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6\" (UID: \"d49b11ba-4f16-4a32-b515-74c95848d2a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:01:06.646734 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.646602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2bb9\" (UniqueName: \"kubernetes.io/projected/2af2bce9-d428-49fb-9b4e-c847785c9541-kube-api-access-l2bb9\") pod \"payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch\" (UID: \"2af2bce9-d428-49fb-9b4e-c847785c9541\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:01:06.647344 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.647321    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d49b11ba-4f16-4a32-b515-74c95848d2a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6\" (UID: \"d49b11ba-4f16-4a32-b515-74c95848d2a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:01:06.647459 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.647340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af2bce9-d428-49fb-9b4e-c847785c9541-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch\" (UID: \"2af2bce9-d428-49fb-9b4e-c847785c9541\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:01:06.654356 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.654334    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6srx\" (UniqueName: \"kubernetes.io/projected/d49b11ba-4f16-4a32-b515-74c95848d2a7-kube-api-access-h6srx\") pod \"payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6\" (UID: \"d49b11ba-4f16-4a32-b515-74c95848d2a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:01:06.654445 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.654340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2bb9\" (UniqueName: \"kubernetes.io/projected/2af2bce9-d428-49fb-9b4e-c847785c9541-kube-api-access-l2bb9\") pod \"payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch\" (UID: \"2af2bce9-d428-49fb-9b4e-c847785c9541\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:01:06.672820 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.672798    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj" event={"ID":"da9a8d93-4f56-46bd-ac4f-eef5fdd90289","Type":"ContainerStarted","Data":"308a2c1cb4796b14fe61e5c4a5420639ecc36a5db1545862be43b0d590d6a4d8"}
Jul 28 21:01:06.771418 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.771397    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:01:06.803656 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.803630    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:01:06.938640 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.938520    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"]
Jul 28 21:01:06.940843 ip-10-0-140-243 kubenswrapper[2575]: W0728 21:01:06.940813    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af2bce9_d428_49fb_9b4e_c847785c9541.slice/crio-f16af263d9bbf663432092e25e0eb0c6c7784d250eb8497bdc5a83574312b89d WatchSource:0}: Error finding container f16af263d9bbf663432092e25e0eb0c6c7784d250eb8497bdc5a83574312b89d: Status 404 returned error can't find the container with id f16af263d9bbf663432092e25e0eb0c6c7784d250eb8497bdc5a83574312b89d
Jul 28 21:01:06.965893 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:06.965864    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"]
Jul 28 21:01:06.968750 ip-10-0-140-243 kubenswrapper[2575]: W0728 21:01:06.968720    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49b11ba_4f16_4a32_b515_74c95848d2a7.slice/crio-18d78da97530220b97aefd335e5f5ae9b152c2ae665d2ef78fc76837853222af WatchSource:0}: Error finding container 18d78da97530220b97aefd335e5f5ae9b152c2ae665d2ef78fc76837853222af: Status 404 returned error can't find the container with id 18d78da97530220b97aefd335e5f5ae9b152c2ae665d2ef78fc76837853222af
Jul 28 21:01:07.679313 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.679275    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6" event={"ID":"d49b11ba-4f16-4a32-b515-74c95848d2a7","Type":"ContainerStarted","Data":"f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63"}
Jul 28 21:01:07.679739 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.679332    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6" event={"ID":"d49b11ba-4f16-4a32-b515-74c95848d2a7","Type":"ContainerStarted","Data":"18d78da97530220b97aefd335e5f5ae9b152c2ae665d2ef78fc76837853222af"}
Jul 28 21:01:07.679739 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.679491    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:01:07.680814 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.680786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch" event={"ID":"2af2bce9-d428-49fb-9b4e-c847785c9541","Type":"ContainerStarted","Data":"fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d"}
Jul 28 21:01:07.680953 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.680834    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch" event={"ID":"2af2bce9-d428-49fb-9b4e-c847785c9541","Type":"ContainerStarted","Data":"f16af263d9bbf663432092e25e0eb0c6c7784d250eb8497bdc5a83574312b89d"}
Jul 28 21:01:07.680953 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.680910    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:01:07.682192 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.682168    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj" event={"ID":"da9a8d93-4f56-46bd-ac4f-eef5fdd90289","Type":"ContainerStarted","Data":"da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2"}
Jul 28 21:01:07.682289 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.682264    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:07.697454 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.697413    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6" podStartSLOduration=1.697400207 podStartE2EDuration="1.697400207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:01:07.695415665 +0000 UTC m=+3167.309652797" watchObservedRunningTime="2026-07-28 21:01:07.697400207 +0000 UTC m=+3167.311637317"
Jul 28 21:01:07.713813 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.713777    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch" podStartSLOduration=1.713765053 podStartE2EDuration="1.713765053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:01:07.713585002 +0000 UTC m=+3167.327822136" watchObservedRunningTime="2026-07-28 21:01:07.713765053 +0000 UTC m=+3167.328002256"
Jul 28 21:01:07.731125 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:07.731087    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj" podStartSLOduration=2.20068498 podStartE2EDuration="2.731074158s" totalImagesPullingTime="530.389178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:01:07.73016856 +0000 UTC m=+3167.344405694" watchObservedRunningTime="2026-07-28 21:01:07.731074158 +0000 UTC m=+3167.345311302"
Jul 28 21:01:08.624311 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:08.624280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:13.693209 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:13.693180    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:01:18.690200 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:18.690168    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:01:18.690720 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:18.690266    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:01:30.676374 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:30.676339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:31.712726 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:31.712686    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:31.814867 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:31.814832    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:31.912135 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:31.912098    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:37.510452 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.510415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:37.615012 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.614981    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"]
Jul 28 21:01:37.619199 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.619173    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:37.619385 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.619360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:37.621510 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.621490    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-542599-serving-cert\""
Jul 28 21:01:37.632111 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.632087    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"]
Jul 28 21:01:37.712105 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.712065    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:37.719919 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.719893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmg57\" (UniqueName: \"kubernetes.io/projected/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-kube-api-access-bmg57\") pod \"maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:37.720009 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.719953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:37.720057 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.720010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-maas-api-tls\") pod \"maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:37.821646 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.821566    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmg57\" (UniqueName: \"kubernetes.io/projected/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-kube-api-access-bmg57\") pod \"maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:37.821786 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.821650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:37.821786 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.821682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-maas-api-tls\") pod \"maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:37.821865 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:01:37.821807    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-serving-cert: secret "maas-api-e2e-ipp-b-542599-serving-cert" not found
Jul 28 21:01:37.821900 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:01:37.821881    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-maas-api-tls podName:aa05c72e-29ca-41bf-bb0f-f5c5106ba615 nodeName:}" failed. No retries permitted until 2026-07-28 21:01:38.321859795 +0000 UTC m=+3197.936096909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-maas-api-tls") pod "maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv" (UID: "aa05c72e-29ca-41bf-bb0f-f5c5106ba615") : secret "maas-api-e2e-ipp-b-542599-serving-cert" not found
Jul 28 21:01:37.822249 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.822215    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:37.845807 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.845777    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmg57\" (UniqueName: \"kubernetes.io/projected/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-kube-api-access-bmg57\") pod \"maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:37.931518 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:37.931487    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:38.326908 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:38.326871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-maas-api-tls\") pod \"maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:38.330804 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:38.330772    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-maas-api-tls\") pod \"maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:38.399886 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:38.399850    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:38.498717 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:38.498683    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:01:38.530418 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:38.530378    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:38.677009 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:38.676975    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"]
Jul 28 21:01:38.678697 ip-10-0-140-243 kubenswrapper[2575]: W0728 21:01:38.678671    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa05c72e_29ca_41bf_bb0f_f5c5106ba615.slice/crio-136bf16c6795b0797a81da9dc0028e268970155c797bba0181e0705402a3b729 WatchSource:0}: Error finding container 136bf16c6795b0797a81da9dc0028e268970155c797bba0181e0705402a3b729: Status 404 returned error can't find the container with id 136bf16c6795b0797a81da9dc0028e268970155c797bba0181e0705402a3b729
Jul 28 21:01:38.838489 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:38.838408    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv" event={"ID":"aa05c72e-29ca-41bf-bb0f-f5c5106ba615","Type":"ContainerStarted","Data":"136bf16c6795b0797a81da9dc0028e268970155c797bba0181e0705402a3b729"}
Jul 28 21:01:39.844410 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:39.844377    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv" event={"ID":"aa05c72e-29ca-41bf-bb0f-f5c5106ba615","Type":"ContainerStarted","Data":"dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8"}
Jul 28 21:01:39.844830 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:39.844503    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:01:39.862203 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:39.862155    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv" podStartSLOduration=2.163582959 podStartE2EDuration="2.862141082s" totalImagesPullingTime="698.558123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:01:39.859522756 +0000 UTC m=+3199.473759890" watchObservedRunningTime="2026-07-28 21:01:39.862141082 +0000 UTC m=+3199.476378215"
Jul 28 21:01:45.854395 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:01:45.854363    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:02:02.544010 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:02:02.543981    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:02:24.877995 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:02:24.877962    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:02:57.122100 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:02:57.122064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:02:57.182851 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:02:57.182815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:27.312870 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:27.312836    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6_d49b11ba-4f16-4a32-b515-74c95848d2a7/payload-processing/0.log"
Jul 28 21:03:28.829925 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:28.829885    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:28.892484 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:28.892448    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:31.193922 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.193887    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:31.355451 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.355423    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:31.422638 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.422608    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"]
Jul 28 21:03:31.426485 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.426445    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:31.429042 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.429012    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-edb657-serving-cert\""
Jul 28 21:03:31.436121 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.436085    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"]
Jul 28 21:03:31.456900 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.456782    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:31.601540 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.601506    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963d9f24-4506-4aa7-8981-a83213193d83-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:31.601540 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.601542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d9f24-4506-4aa7-8981-a83213193d83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:31.601809 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.601578    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmjkj\" (UniqueName: \"kubernetes.io/projected/963d9f24-4506-4aa7-8981-a83213193d83-kube-api-access-fmjkj\") pod \"maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:31.702054 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.702023    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmjkj\" (UniqueName: \"kubernetes.io/projected/963d9f24-4506-4aa7-8981-a83213193d83-kube-api-access-fmjkj\") pod \"maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:31.702269 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.702129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963d9f24-4506-4aa7-8981-a83213193d83-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:31.702269 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.702151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d9f24-4506-4aa7-8981-a83213193d83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:31.702411 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:03:31.702269    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-serving-cert: secret "maas-api-e2e-ipp-cleanup-edb657-serving-cert" not found
Jul 28 21:03:31.702411 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:03:31.702344    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/963d9f24-4506-4aa7-8981-a83213193d83-maas-api-tls podName:963d9f24-4506-4aa7-8981-a83213193d83 nodeName:}" failed. No retries permitted until 2026-07-28 21:03:32.20232539 +0000 UTC m=+3311.816562503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/963d9f24-4506-4aa7-8981-a83213193d83-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj" (UID: "963d9f24-4506-4aa7-8981-a83213193d83") : secret "maas-api-e2e-ipp-cleanup-edb657-serving-cert" not found
Jul 28 21:03:31.702872 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.702853    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963d9f24-4506-4aa7-8981-a83213193d83-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:31.712893 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.712832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmjkj\" (UniqueName: \"kubernetes.io/projected/963d9f24-4506-4aa7-8981-a83213193d83-kube-api-access-fmjkj\") pod \"maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:31.846278 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:31.846244    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:32.097137 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:32.097038    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:32.202309 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:32.202270    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:03:32.207626 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:32.207588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d9f24-4506-4aa7-8981-a83213193d83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:32.210255 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:32.210211    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d9f24-4506-4aa7-8981-a83213193d83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:32.340381 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:32.340342    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:32.482501 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:32.482467    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"]
Jul 28 21:03:32.485456 ip-10-0-140-243 kubenswrapper[2575]: W0728 21:03:32.485423    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963d9f24_4506_4aa7_8981_a83213193d83.slice/crio-c6c0be96d891fa7cb7dbea661eeaa48b1f720d82769739e9b614a6a128882cfb WatchSource:0}: Error finding container c6c0be96d891fa7cb7dbea661eeaa48b1f720d82769739e9b614a6a128882cfb: Status 404 returned error can't find the container with id c6c0be96d891fa7cb7dbea661eeaa48b1f720d82769739e9b614a6a128882cfb
Jul 28 21:03:33.410308 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:33.410271    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj" event={"ID":"963d9f24-4506-4aa7-8981-a83213193d83","Type":"ContainerStarted","Data":"5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3"}
Jul 28 21:03:33.410776 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:33.410330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj" event={"ID":"963d9f24-4506-4aa7-8981-a83213193d83","Type":"ContainerStarted","Data":"c6c0be96d891fa7cb7dbea661eeaa48b1f720d82769739e9b614a6a128882cfb"}
Jul 28 21:03:33.410776 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:33.410357    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:33.426101 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:33.426056    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj" podStartSLOduration=1.840529385 podStartE2EDuration="2.426042391s" totalImagesPullingTime="585.513006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:03:33.425534037 +0000 UTC m=+3313.039771183" watchObservedRunningTime="2026-07-28 21:03:33.426042391 +0000 UTC m=+3313.040279524"
Jul 28 21:03:39.422317 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:39.422285    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:03:56.142435 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:03:56.142352    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:01.891170 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:01.891135    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"]
Jul 28 21:04:01.891742 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:01.891399    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj" podUID="963d9f24-4506-4aa7-8981-a83213193d83" containerName="maas-api" containerID="cri-o://5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3" gracePeriod=30
Jul 28 21:04:01.913254 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:01.913213    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:01.978579 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:01.978547    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:02.085558 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.085519    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:02.155600 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.155575    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:04:02.156247 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.156203    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:02.265677 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.265646    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d9f24-4506-4aa7-8981-a83213193d83-maas-api-tls\") pod \"963d9f24-4506-4aa7-8981-a83213193d83\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") "
Jul 28 21:04:02.265840 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.265747    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963d9f24-4506-4aa7-8981-a83213193d83-openshift-service-ca\") pod \"963d9f24-4506-4aa7-8981-a83213193d83\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") "
Jul 28 21:04:02.265840 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.265815    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmjkj\" (UniqueName: \"kubernetes.io/projected/963d9f24-4506-4aa7-8981-a83213193d83-kube-api-access-fmjkj\") pod \"963d9f24-4506-4aa7-8981-a83213193d83\" (UID: \"963d9f24-4506-4aa7-8981-a83213193d83\") "
Jul 28 21:04:02.266212 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.266172    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963d9f24-4506-4aa7-8981-a83213193d83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "963d9f24-4506-4aa7-8981-a83213193d83" (UID: "963d9f24-4506-4aa7-8981-a83213193d83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:04:02.267862 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.267825    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963d9f24-4506-4aa7-8981-a83213193d83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "963d9f24-4506-4aa7-8981-a83213193d83" (UID: "963d9f24-4506-4aa7-8981-a83213193d83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:04:02.267862 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.267850    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963d9f24-4506-4aa7-8981-a83213193d83-kube-api-access-fmjkj" (OuterVolumeSpecName: "kube-api-access-fmjkj") pod "963d9f24-4506-4aa7-8981-a83213193d83" (UID: "963d9f24-4506-4aa7-8981-a83213193d83"). InnerVolumeSpecName "kube-api-access-fmjkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:04:02.366466 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.366429    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmjkj\" (UniqueName: \"kubernetes.io/projected/963d9f24-4506-4aa7-8981-a83213193d83-kube-api-access-fmjkj\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:04:02.366466 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.366460    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963d9f24-4506-4aa7-8981-a83213193d83-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:04:02.366466 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.366469    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963d9f24-4506-4aa7-8981-a83213193d83-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:04:02.458171 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.458093    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:02.561132 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.561100    2575 generic.go:356] "Generic (PLEG): container finished" podID="963d9f24-4506-4aa7-8981-a83213193d83" containerID="5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3" exitCode=0
Jul 28 21:04:02.561331 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.561154    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"
Jul 28 21:04:02.561331 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.561186    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj" event={"ID":"963d9f24-4506-4aa7-8981-a83213193d83","Type":"ContainerDied","Data":"5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3"}
Jul 28 21:04:02.561331 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.561263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj" event={"ID":"963d9f24-4506-4aa7-8981-a83213193d83","Type":"ContainerDied","Data":"c6c0be96d891fa7cb7dbea661eeaa48b1f720d82769739e9b614a6a128882cfb"}
Jul 28 21:04:02.561331 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.561280    2575 scope.go:117] "RemoveContainer" containerID="5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3"
Jul 28 21:04:02.570132 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.570087    2575 scope.go:117] "RemoveContainer" containerID="5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3"
Jul 28 21:04:02.570382 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:04:02.570361    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3\": container with ID starting with 5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3 not found: ID does not exist" containerID="5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3"
Jul 28 21:04:02.570455 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.570390    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3"} err="failed to get container status \"5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3\": rpc error: code = NotFound desc = could not find container \"5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3\": container with ID starting with 5829fb3687d9510fdca2f9e36f5f334a54db47568515b73a3a094c2b10306ad3 not found: ID does not exist"
Jul 28 21:04:02.585014 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.584991    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"]
Jul 28 21:04:02.588750 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.588730    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-edb657-66459c8c57-97snj"]
Jul 28 21:04:02.836786 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:02.836697    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="963d9f24-4506-4aa7-8981-a83213193d83" path="/var/lib/kubelet/pods/963d9f24-4506-4aa7-8981-a83213193d83/volumes"
Jul 28 21:04:08.520126 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:08.520092    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:08.620028 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:08.619994    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:08.784155 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:08.784079    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:09.045459 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:09.045379    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:27.953467 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:27.953427    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"]
Jul 28 21:04:27.953958 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:27.953701    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj" podUID="da9a8d93-4f56-46bd-ac4f-eef5fdd90289" containerName="maas-api" containerID="cri-o://da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2" gracePeriod=30
Jul 28 21:04:27.978991 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:27.978951    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:27.996321 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:27.996291    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"]
Jul 28 21:04:27.996549 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:27.996524    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6" podUID="d49b11ba-4f16-4a32-b515-74c95848d2a7" containerName="payload-processing" containerID="cri-o://f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63" gracePeriod=30
Jul 28 21:04:28.015485 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.015452    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"]
Jul 28 21:04:28.015736 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.015711    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch" podUID="2af2bce9-d428-49fb-9b4e-c847785c9541" containerName="payload-pre-processing" containerID="cri-o://fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d" gracePeriod=30
Jul 28 21:04:28.079248 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.079188    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:28.180979 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.180941    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:28.290101 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.290074    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:04:28.316082 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.316058    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:04:28.326794 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.326769    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:04:28.403037 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.403004    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af2bce9-d428-49fb-9b4e-c847785c9541-plugins-config-volume\") pod \"2af2bce9-d428-49fb-9b4e-c847785c9541\" (UID: \"2af2bce9-d428-49fb-9b4e-c847785c9541\") "
Jul 28 21:04:28.403216 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.403058    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-openshift-service-ca\") pod \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") "
Jul 28 21:04:28.403216 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.403145    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k467q\" (UniqueName: \"kubernetes.io/projected/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-kube-api-access-k467q\") pod \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") "
Jul 28 21:04:28.403216 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.403174    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2bb9\" (UniqueName: \"kubernetes.io/projected/2af2bce9-d428-49fb-9b4e-c847785c9541-kube-api-access-l2bb9\") pod \"2af2bce9-d428-49fb-9b4e-c847785c9541\" (UID: \"2af2bce9-d428-49fb-9b4e-c847785c9541\") "
Jul 28 21:04:28.403401 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.403218    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-maas-api-tls\") pod \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\" (UID: \"da9a8d93-4f56-46bd-ac4f-eef5fdd90289\") "
Jul 28 21:04:28.403493 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.403472    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af2bce9-d428-49fb-9b4e-c847785c9541-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2af2bce9-d428-49fb-9b4e-c847785c9541" (UID: "2af2bce9-d428-49fb-9b4e-c847785c9541"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:04:28.403553 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.403513    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da9a8d93-4f56-46bd-ac4f-eef5fdd90289" (UID: "da9a8d93-4f56-46bd-ac4f-eef5fdd90289"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:04:28.403596 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.403555    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af2bce9-d428-49fb-9b4e-c847785c9541-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:04:28.405469 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.405443    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-kube-api-access-k467q" (OuterVolumeSpecName: "kube-api-access-k467q") pod "da9a8d93-4f56-46bd-ac4f-eef5fdd90289" (UID: "da9a8d93-4f56-46bd-ac4f-eef5fdd90289"). InnerVolumeSpecName "kube-api-access-k467q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:04:28.405573 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.405474    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af2bce9-d428-49fb-9b4e-c847785c9541-kube-api-access-l2bb9" (OuterVolumeSpecName: "kube-api-access-l2bb9") pod "2af2bce9-d428-49fb-9b4e-c847785c9541" (UID: "2af2bce9-d428-49fb-9b4e-c847785c9541"). InnerVolumeSpecName "kube-api-access-l2bb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:04:28.405573 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.405477    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da9a8d93-4f56-46bd-ac4f-eef5fdd90289" (UID: "da9a8d93-4f56-46bd-ac4f-eef5fdd90289"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:04:28.504422 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.504345    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6srx\" (UniqueName: \"kubernetes.io/projected/d49b11ba-4f16-4a32-b515-74c95848d2a7-kube-api-access-h6srx\") pod \"d49b11ba-4f16-4a32-b515-74c95848d2a7\" (UID: \"d49b11ba-4f16-4a32-b515-74c95848d2a7\") "
Jul 28 21:04:28.504422 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.504377    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d49b11ba-4f16-4a32-b515-74c95848d2a7-plugins-config-volume\") pod \"d49b11ba-4f16-4a32-b515-74c95848d2a7\" (UID: \"d49b11ba-4f16-4a32-b515-74c95848d2a7\") "
Jul 28 21:04:28.504713 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.504695    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49b11ba-4f16-4a32-b515-74c95848d2a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d49b11ba-4f16-4a32-b515-74c95848d2a7" (UID: "d49b11ba-4f16-4a32-b515-74c95848d2a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:04:28.504786 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.504767    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k467q\" (UniqueName: \"kubernetes.io/projected/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-kube-api-access-k467q\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:04:28.504826 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.504790    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2bb9\" (UniqueName: \"kubernetes.io/projected/2af2bce9-d428-49fb-9b4e-c847785c9541-kube-api-access-l2bb9\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:04:28.504826 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.504801    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:04:28.504826 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.504810    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9a8d93-4f56-46bd-ac4f-eef5fdd90289-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:04:28.506396 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.506378    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49b11ba-4f16-4a32-b515-74c95848d2a7-kube-api-access-h6srx" (OuterVolumeSpecName: "kube-api-access-h6srx") pod "d49b11ba-4f16-4a32-b515-74c95848d2a7" (UID: "d49b11ba-4f16-4a32-b515-74c95848d2a7"). InnerVolumeSpecName "kube-api-access-h6srx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:04:28.551861 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.551832    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:04:28.605310 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.605278    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6srx\" (UniqueName: \"kubernetes.io/projected/d49b11ba-4f16-4a32-b515-74c95848d2a7-kube-api-access-h6srx\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:04:28.605310 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.605302    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d49b11ba-4f16-4a32-b515-74c95848d2a7-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:04:28.705538 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.705502    2575 generic.go:356] "Generic (PLEG): container finished" podID="d49b11ba-4f16-4a32-b515-74c95848d2a7" containerID="f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63" exitCode=0
Jul 28 21:04:28.705733 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.705559    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"
Jul 28 21:04:28.705733 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.705586    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6" event={"ID":"d49b11ba-4f16-4a32-b515-74c95848d2a7","Type":"ContainerDied","Data":"f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63"}
Jul 28 21:04:28.705733 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.705644    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6" event={"ID":"d49b11ba-4f16-4a32-b515-74c95848d2a7","Type":"ContainerDied","Data":"18d78da97530220b97aefd335e5f5ae9b152c2ae665d2ef78fc76837853222af"}
Jul 28 21:04:28.705733 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.705677    2575 scope.go:117] "RemoveContainer" containerID="f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63"
Jul 28 21:04:28.707067 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.707033    2575 generic.go:356] "Generic (PLEG): container finished" podID="2af2bce9-d428-49fb-9b4e-c847785c9541" containerID="fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d" exitCode=0
Jul 28 21:04:28.707195 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.707081    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"
Jul 28 21:04:28.707195 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.707076    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch" event={"ID":"2af2bce9-d428-49fb-9b4e-c847785c9541","Type":"ContainerDied","Data":"fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d"}
Jul 28 21:04:28.707195 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.707182    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch" event={"ID":"2af2bce9-d428-49fb-9b4e-c847785c9541","Type":"ContainerDied","Data":"f16af263d9bbf663432092e25e0eb0c6c7784d250eb8497bdc5a83574312b89d"}
Jul 28 21:04:28.708640 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.708616    2575 generic.go:356] "Generic (PLEG): container finished" podID="da9a8d93-4f56-46bd-ac4f-eef5fdd90289" containerID="da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2" exitCode=0
Jul 28 21:04:28.708769 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.708649    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"
Jul 28 21:04:28.708769 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.708687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj" event={"ID":"da9a8d93-4f56-46bd-ac4f-eef5fdd90289","Type":"ContainerDied","Data":"da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2"}
Jul 28 21:04:28.708769 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.708723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj" event={"ID":"da9a8d93-4f56-46bd-ac4f-eef5fdd90289","Type":"ContainerDied","Data":"308a2c1cb4796b14fe61e5c4a5420639ecc36a5db1545862be43b0d590d6a4d8"}
Jul 28 21:04:28.716579 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.716561    2575 scope.go:117] "RemoveContainer" containerID="f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63"
Jul 28 21:04:28.716862 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:04:28.716839    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63\": container with ID starting with f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63 not found: ID does not exist" containerID="f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63"
Jul 28 21:04:28.716931 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.716875    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63"} err="failed to get container status \"f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63\": rpc error: code = NotFound desc = could not find container \"f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63\": container with ID starting with f7962831a3f7d2a220a87abd88bf1122ab642c9a458e4fabf01a345fdb7bdd63 not found: ID does not exist"
Jul 28 21:04:28.716931 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.716910    2575 scope.go:117] "RemoveContainer" containerID="fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d"
Jul 28 21:04:28.725110 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.725087    2575 scope.go:117] "RemoveContainer" containerID="fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d"
Jul 28 21:04:28.725389 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:04:28.725373    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d\": container with ID starting with fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d not found: ID does not exist" containerID="fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d"
Jul 28 21:04:28.725443 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.725397    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d"} err="failed to get container status \"fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d\": rpc error: code = NotFound desc = could not find container \"fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d\": container with ID starting with fa701fcdbc42b48fc47afa6e92a53510d3282bd2bc4e0e4b2e9815477bc3f81d not found: ID does not exist"
Jul 28 21:04:28.725443 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.725412    2575 scope.go:117] "RemoveContainer" containerID="da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2"
Jul 28 21:04:28.731854 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.731829    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"]
Jul 28 21:04:28.733985 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.733967    2575 scope.go:117] "RemoveContainer" containerID="da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2"
Jul 28 21:04:28.734654 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:04:28.734412    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2\": container with ID starting with da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2 not found: ID does not exist" containerID="da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2"
Jul 28 21:04:28.734654 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.734449    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2"} err="failed to get container status \"da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2\": rpc error: code = NotFound desc = could not find container \"da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2\": container with ID starting with da834831fcc9cd09b7de31ecbbc17edf33eee31695c17ad56f203b61d601a9d2 not found: ID does not exist"
Jul 28 21:04:28.736346 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.736319    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce337a-6cd49bd6b-xfxc6"]
Jul 28 21:04:28.747140 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.747119    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"]
Jul 28 21:04:28.750847 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.750825    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce337a-7bf9b55fb7-b2tsj"]
Jul 28 21:04:28.760550 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.760488    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"]
Jul 28 21:04:28.767588 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.767570    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce337a-c9768c88-5jvch"]
Jul 28 21:04:28.836994 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.836962    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af2bce9-d428-49fb-9b4e-c847785c9541" path="/var/lib/kubelet/pods/2af2bce9-d428-49fb-9b4e-c847785c9541/volumes"
Jul 28 21:04:28.837365 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.837351    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49b11ba-4f16-4a32-b515-74c95848d2a7" path="/var/lib/kubelet/pods/d49b11ba-4f16-4a32-b515-74c95848d2a7/volumes"
Jul 28 21:04:28.837711 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:28.837700    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da9a8d93-4f56-46bd-ac4f-eef5fdd90289" path="/var/lib/kubelet/pods/da9a8d93-4f56-46bd-ac4f-eef5fdd90289/volumes"
Jul 28 21:04:33.624820 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:04:33.624785    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:10.122683 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:10.122649    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:10.950479 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:10.950448    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:16.446814 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:16.446720    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:16.513743 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:16.513708    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:16.756988 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:16.756895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:16.981044 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:16.981015    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:22.219689 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:22.219652    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:28.229684 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.229198    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"]
Jul 28 21:05:28.240707 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.240626    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv" podUID="aa05c72e-29ca-41bf-bb0f-f5c5106ba615" containerName="maas-api" containerID="cri-o://dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8" gracePeriod=30
Jul 28 21:05:28.246358 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.246333    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:28.328135 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.328100    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:28.394691 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.394656    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:28.457844 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.457795    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:28.501786 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.501761    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:05:28.544090 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.544054    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-openshift-service-ca\") pod \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") "
Jul 28 21:05:28.544285 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.544115    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-maas-api-tls\") pod \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") "
Jul 28 21:05:28.544285 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.544179    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmg57\" (UniqueName: \"kubernetes.io/projected/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-kube-api-access-bmg57\") pod \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\" (UID: \"aa05c72e-29ca-41bf-bb0f-f5c5106ba615\") "
Jul 28 21:05:28.544507 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.544486    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa05c72e-29ca-41bf-bb0f-f5c5106ba615" (UID: "aa05c72e-29ca-41bf-bb0f-f5c5106ba615"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:05:28.546327 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.546294    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa05c72e-29ca-41bf-bb0f-f5c5106ba615" (UID: "aa05c72e-29ca-41bf-bb0f-f5c5106ba615"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:05:28.546502 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.546479    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-kube-api-access-bmg57" (OuterVolumeSpecName: "kube-api-access-bmg57") pod "aa05c72e-29ca-41bf-bb0f-f5c5106ba615" (UID: "aa05c72e-29ca-41bf-bb0f-f5c5106ba615"). InnerVolumeSpecName "kube-api-access-bmg57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:05:28.645849 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.645813    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:05:28.645849 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.645843    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:05:28.645849 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.645852    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmg57\" (UniqueName: \"kubernetes.io/projected/aa05c72e-29ca-41bf-bb0f-f5c5106ba615-kube-api-access-bmg57\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:05:28.840516 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:28.840408    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:29.021057 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:29.021028    2575 generic.go:356] "Generic (PLEG): container finished" podID="aa05c72e-29ca-41bf-bb0f-f5c5106ba615" containerID="dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8" exitCode=0
Jul 28 21:05:29.021256 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:29.021075    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"
Jul 28 21:05:29.021256 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:29.021109    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv" event={"ID":"aa05c72e-29ca-41bf-bb0f-f5c5106ba615","Type":"ContainerDied","Data":"dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8"}
Jul 28 21:05:29.021256 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:29.021167    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv" event={"ID":"aa05c72e-29ca-41bf-bb0f-f5c5106ba615","Type":"ContainerDied","Data":"136bf16c6795b0797a81da9dc0028e268970155c797bba0181e0705402a3b729"}
Jul 28 21:05:29.021256 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:29.021193    2575 scope.go:117] "RemoveContainer" containerID="dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8"
Jul 28 21:05:29.030939 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:29.030920    2575 scope.go:117] "RemoveContainer" containerID="dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8"
Jul 28 21:05:29.031341 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:05:29.031313    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8\": container with ID starting with dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8 not found: ID does not exist" containerID="dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8"
Jul 28 21:05:29.031466 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:29.031351    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8"} err="failed to get container status \"dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8\": rpc error: code = NotFound desc = could not find container \"dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8\": container with ID starting with dc21a23617599ec92766433da5c055549db46792cc7e1e6a9cb8d4f31bfdcaa8 not found: ID does not exist"
Jul 28 21:05:29.036165 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:29.036140    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"]
Jul 28 21:05:29.040142 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:29.040118    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-542599-5d74c77496-8wkdv"]
Jul 28 21:05:30.837054 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:30.837018    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa05c72e-29ca-41bf-bb0f-f5c5106ba615" path="/var/lib/kubelet/pods/aa05c72e-29ca-41bf-bb0f-f5c5106ba615/volumes"
Jul 28 21:05:32.111825 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:32.111775    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:34.740716 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:34.740685    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:34.892069 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:34.892040    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:35.012181 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:35.012104    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:35.296958 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:35.296874    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:45.823907 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:45.823870    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"]
Jul 28 21:05:45.824865 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:45.824805    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s" podUID="5c1d6a7f-2f16-4cbe-add6-375b70062942" containerName="maas-api" containerID="cri-o://162007bef538619d0c4e7370dd383eb45def8ad0c346e1c40b655c602e2d2c1f" gracePeriod=30
Jul 28 21:05:45.853331 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:45.853305    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:45.892314 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:45.892267    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"]
Jul 28 21:05:45.892574 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:45.892544    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x" podUID="ddbfd028-8b80-472d-a326-2a5cbe95ee57" containerName="payload-pre-processing" containerID="cri-o://9e08887ebb672e258c82f2543183955188ea02c72523e9840ae78364fc6ad38b" gracePeriod=30
Jul 28 21:05:45.952383 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:45.952354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:46.035897 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.035864    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:46.093335 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.093249    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:46.107146 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.107115    2575 generic.go:356] "Generic (PLEG): container finished" podID="5c1d6a7f-2f16-4cbe-add6-375b70062942" containerID="162007bef538619d0c4e7370dd383eb45def8ad0c346e1c40b655c602e2d2c1f" exitCode=0
Jul 28 21:05:46.107309 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.107295    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s" event={"ID":"5c1d6a7f-2f16-4cbe-add6-375b70062942","Type":"ContainerDied","Data":"162007bef538619d0c4e7370dd383eb45def8ad0c346e1c40b655c602e2d2c1f"}
Jul 28 21:05:46.109274 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.109249    2575 generic.go:356] "Generic (PLEG): container finished" podID="ddbfd028-8b80-472d-a326-2a5cbe95ee57" containerID="9e08887ebb672e258c82f2543183955188ea02c72523e9840ae78364fc6ad38b" exitCode=0
Jul 28 21:05:46.109423 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.109318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x" event={"ID":"ddbfd028-8b80-472d-a326-2a5cbe95ee57","Type":"ContainerDied","Data":"9e08887ebb672e258c82f2543183955188ea02c72523e9840ae78364fc6ad38b"}
Jul 28 21:05:46.147995 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.147973    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 21:05:46.179898 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.179877    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 21:05:46.308127 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.308097    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbfd028-8b80-472d-a326-2a5cbe95ee57-plugins-config-volume\") pod \"ddbfd028-8b80-472d-a326-2a5cbe95ee57\" (UID: \"ddbfd028-8b80-472d-a326-2a5cbe95ee57\") "
Jul 28 21:05:46.308320 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.308136    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c1d6a7f-2f16-4cbe-add6-375b70062942-maas-api-tls\") pod \"5c1d6a7f-2f16-4cbe-add6-375b70062942\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") "
Jul 28 21:05:46.308320 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.308168    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drbs5\" (UniqueName: \"kubernetes.io/projected/5c1d6a7f-2f16-4cbe-add6-375b70062942-kube-api-access-drbs5\") pod \"5c1d6a7f-2f16-4cbe-add6-375b70062942\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") "
Jul 28 21:05:46.308320 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.308205    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1d6a7f-2f16-4cbe-add6-375b70062942-openshift-service-ca\") pod \"5c1d6a7f-2f16-4cbe-add6-375b70062942\" (UID: \"5c1d6a7f-2f16-4cbe-add6-375b70062942\") "
Jul 28 21:05:46.308320 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.308282    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcqjl\" (UniqueName: \"kubernetes.io/projected/ddbfd028-8b80-472d-a326-2a5cbe95ee57-kube-api-access-fcqjl\") pod \"ddbfd028-8b80-472d-a326-2a5cbe95ee57\" (UID: \"ddbfd028-8b80-472d-a326-2a5cbe95ee57\") "
Jul 28 21:05:46.308605 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.308575    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbfd028-8b80-472d-a326-2a5cbe95ee57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ddbfd028-8b80-472d-a326-2a5cbe95ee57" (UID: "ddbfd028-8b80-472d-a326-2a5cbe95ee57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:05:46.308671 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.308600    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1d6a7f-2f16-4cbe-add6-375b70062942-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c1d6a7f-2f16-4cbe-add6-375b70062942" (UID: "5c1d6a7f-2f16-4cbe-add6-375b70062942"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:05:46.310396 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.310364    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c1d6a7f-2f16-4cbe-add6-375b70062942-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c1d6a7f-2f16-4cbe-add6-375b70062942" (UID: "5c1d6a7f-2f16-4cbe-add6-375b70062942"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:05:46.310507 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.310404    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbfd028-8b80-472d-a326-2a5cbe95ee57-kube-api-access-fcqjl" (OuterVolumeSpecName: "kube-api-access-fcqjl") pod "ddbfd028-8b80-472d-a326-2a5cbe95ee57" (UID: "ddbfd028-8b80-472d-a326-2a5cbe95ee57"). InnerVolumeSpecName "kube-api-access-fcqjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:05:46.310507 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.310481    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1d6a7f-2f16-4cbe-add6-375b70062942-kube-api-access-drbs5" (OuterVolumeSpecName: "kube-api-access-drbs5") pod "5c1d6a7f-2f16-4cbe-add6-375b70062942" (UID: "5c1d6a7f-2f16-4cbe-add6-375b70062942"). InnerVolumeSpecName "kube-api-access-drbs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:05:46.409405 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.409376    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbfd028-8b80-472d-a326-2a5cbe95ee57-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:05:46.409405 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.409404    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c1d6a7f-2f16-4cbe-add6-375b70062942-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:05:46.409586 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.409414    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbs5\" (UniqueName: \"kubernetes.io/projected/5c1d6a7f-2f16-4cbe-add6-375b70062942-kube-api-access-drbs5\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:05:46.409586 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.409423    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1d6a7f-2f16-4cbe-add6-375b70062942-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:05:46.409586 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.409431    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcqjl\" (UniqueName: \"kubernetes.io/projected/ddbfd028-8b80-472d-a326-2a5cbe95ee57-kube-api-access-fcqjl\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:05:46.428780 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:46.428719    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:47.116151 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:47.116060    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"
Jul 28 21:05:47.116151 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:47.116099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s" event={"ID":"5c1d6a7f-2f16-4cbe-add6-375b70062942","Type":"ContainerDied","Data":"3240592f7c4c9b4b2e4fea691a7a3f59fc89d69365332091840ecdac115c2d57"}
Jul 28 21:05:47.116715 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:47.116171    2575 scope.go:117] "RemoveContainer" containerID="162007bef538619d0c4e7370dd383eb45def8ad0c346e1c40b655c602e2d2c1f"
Jul 28 21:05:47.117616 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:47.117526    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x" event={"ID":"ddbfd028-8b80-472d-a326-2a5cbe95ee57","Type":"ContainerDied","Data":"1ad314f1677dfa3e27c0c2b79f5d360ffb9960643944c499ed6055642985b82a"}
Jul 28 21:05:47.117616 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:47.117554    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"
Jul 28 21:05:47.127038 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:47.127021    2575 scope.go:117] "RemoveContainer" containerID="9e08887ebb672e258c82f2543183955188ea02c72523e9840ae78364fc6ad38b"
Jul 28 21:05:47.136254 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:47.136217    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"]
Jul 28 21:05:47.139727 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:47.139708    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-cc370c-7758b6458d-zvg4x"]
Jul 28 21:05:47.149652 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:47.149627    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"]
Jul 28 21:05:47.153579 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:47.153549    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cc370c-bd8c48fdd-5tw7s"]
Jul 28 21:05:48.835533 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:48.835495    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c1d6a7f-2f16-4cbe-add6-375b70062942" path="/var/lib/kubelet/pods/5c1d6a7f-2f16-4cbe-add6-375b70062942/volumes"
Jul 28 21:05:48.836022 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:48.835835    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddbfd028-8b80-472d-a326-2a5cbe95ee57" path="/var/lib/kubelet/pods/ddbfd028-8b80-472d-a326-2a5cbe95ee57/volumes"
Jul 28 21:05:52.427079 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:52.427043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:52.561590 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:52.561562    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:52.743834 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:52.743739    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:52.834981 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:52.834948    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:05:52.937545 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:05:52.937514    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:06:03.466133 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.466097    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"]
Jul 28 21:06:03.468736 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.466366    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t" podUID="08c14450-d207-40d6-8e7c-4bb0140145c7" containerName="maas-api" containerID="cri-o://6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6" gracePeriod=30
Jul 28 21:06:03.490572 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.490538    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:06:03.523303 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.523199    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"]
Jul 28 21:06:03.523520 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.523463    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j" podUID="8b479785-71d5-4bd2-9445-4780b792ca49" containerName="payload-pre-processing" containerID="cri-o://39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2" gracePeriod=30
Jul 28 21:06:03.564562 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.564535    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:06:03.672011 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.671978    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:06:03.731540 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.731505    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:06:03.774051 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.774025    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 21:06:03.804971 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.804945    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 21:06:03.961151 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.961118    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm6fh\" (UniqueName: \"kubernetes.io/projected/08c14450-d207-40d6-8e7c-4bb0140145c7-kube-api-access-vm6fh\") pod \"08c14450-d207-40d6-8e7c-4bb0140145c7\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") "
Jul 28 21:06:03.961151 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.961151    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c14450-d207-40d6-8e7c-4bb0140145c7-openshift-service-ca\") pod \"08c14450-d207-40d6-8e7c-4bb0140145c7\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") "
Jul 28 21:06:03.961381 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.961170    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kcq9\" (UniqueName: \"kubernetes.io/projected/8b479785-71d5-4bd2-9445-4780b792ca49-kube-api-access-9kcq9\") pod \"8b479785-71d5-4bd2-9445-4780b792ca49\" (UID: \"8b479785-71d5-4bd2-9445-4780b792ca49\") "
Jul 28 21:06:03.961381 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.961219    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b479785-71d5-4bd2-9445-4780b792ca49-plugins-config-volume\") pod \"8b479785-71d5-4bd2-9445-4780b792ca49\" (UID: \"8b479785-71d5-4bd2-9445-4780b792ca49\") "
Jul 28 21:06:03.961381 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.961308    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c14450-d207-40d6-8e7c-4bb0140145c7-maas-api-tls\") pod \"08c14450-d207-40d6-8e7c-4bb0140145c7\" (UID: \"08c14450-d207-40d6-8e7c-4bb0140145c7\") "
Jul 28 21:06:03.961870 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.961548    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c14450-d207-40d6-8e7c-4bb0140145c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08c14450-d207-40d6-8e7c-4bb0140145c7" (UID: "08c14450-d207-40d6-8e7c-4bb0140145c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:06:03.961870 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.961673    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b479785-71d5-4bd2-9445-4780b792ca49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b479785-71d5-4bd2-9445-4780b792ca49" (UID: "8b479785-71d5-4bd2-9445-4780b792ca49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:06:03.961870 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.961743    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c14450-d207-40d6-8e7c-4bb0140145c7-openshift-service-ca\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:06:03.963396 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.963371    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c14450-d207-40d6-8e7c-4bb0140145c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08c14450-d207-40d6-8e7c-4bb0140145c7" (UID: "08c14450-d207-40d6-8e7c-4bb0140145c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:06:03.963497 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.963450    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b479785-71d5-4bd2-9445-4780b792ca49-kube-api-access-9kcq9" (OuterVolumeSpecName: "kube-api-access-9kcq9") pod "8b479785-71d5-4bd2-9445-4780b792ca49" (UID: "8b479785-71d5-4bd2-9445-4780b792ca49"). InnerVolumeSpecName "kube-api-access-9kcq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:06:03.963497 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:03.963477    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c14450-d207-40d6-8e7c-4bb0140145c7-kube-api-access-vm6fh" (OuterVolumeSpecName: "kube-api-access-vm6fh") pod "08c14450-d207-40d6-8e7c-4bb0140145c7" (UID: "08c14450-d207-40d6-8e7c-4bb0140145c7"). InnerVolumeSpecName "kube-api-access-vm6fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:06:04.041769 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.041695    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:06:04.062855 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.062832    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c14450-d207-40d6-8e7c-4bb0140145c7-maas-api-tls\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:06:04.062855 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.062855    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm6fh\" (UniqueName: \"kubernetes.io/projected/08c14450-d207-40d6-8e7c-4bb0140145c7-kube-api-access-vm6fh\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:06:04.062965 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.062866    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kcq9\" (UniqueName: \"kubernetes.io/projected/8b479785-71d5-4bd2-9445-4780b792ca49-kube-api-access-9kcq9\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:06:04.062965 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.062875    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b479785-71d5-4bd2-9445-4780b792ca49-plugins-config-volume\") on node \"ip-10-0-140-243.ec2.internal\" DevicePath \"\""
Jul 28 21:06:04.210388 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.210350    2575 generic.go:356] "Generic (PLEG): container finished" podID="8b479785-71d5-4bd2-9445-4780b792ca49" containerID="39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2" exitCode=0
Jul 28 21:06:04.210592 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.210415    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"
Jul 28 21:06:04.210592 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.210430    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j" event={"ID":"8b479785-71d5-4bd2-9445-4780b792ca49","Type":"ContainerDied","Data":"39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2"}
Jul 28 21:06:04.210592 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.210513    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j" event={"ID":"8b479785-71d5-4bd2-9445-4780b792ca49","Type":"ContainerDied","Data":"8ced87a3dd881dae936cdd8278a97c728736f8ae2c229d1010725ae3ce7dfb5a"}
Jul 28 21:06:04.210592 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.210537    2575 scope.go:117] "RemoveContainer" containerID="39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2"
Jul 28 21:06:04.211961 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.211936    2575 generic.go:356] "Generic (PLEG): container finished" podID="08c14450-d207-40d6-8e7c-4bb0140145c7" containerID="6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6" exitCode=0
Jul 28 21:06:04.212087 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.211988    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"
Jul 28 21:06:04.212087 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.212016    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t" event={"ID":"08c14450-d207-40d6-8e7c-4bb0140145c7","Type":"ContainerDied","Data":"6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6"}
Jul 28 21:06:04.212087 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.212066    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t" event={"ID":"08c14450-d207-40d6-8e7c-4bb0140145c7","Type":"ContainerDied","Data":"59fcaeb46d55293cca5bd36b25cb91959672ef32598609a15695b2a92ff34992"}
Jul 28 21:06:04.219895 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.219878    2575 scope.go:117] "RemoveContainer" containerID="39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2"
Jul 28 21:06:04.220339 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:06:04.220142    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2\": container with ID starting with 39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2 not found: ID does not exist" containerID="39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2"
Jul 28 21:06:04.220339 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.220166    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2"} err="failed to get container status \"39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2\": rpc error: code = NotFound desc = could not find container \"39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2\": container with ID starting with 39ef5814ec6c8a3b062df63fe0d243c41dd17138e912c252b5d5687181aa2af2 not found: ID does not exist"
Jul 28 21:06:04.220339 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.220182    2575 scope.go:117] "RemoveContainer" containerID="6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6"
Jul 28 21:06:04.228884 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.228865    2575 scope.go:117] "RemoveContainer" containerID="6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6"
Jul 28 21:06:04.229157 ip-10-0-140-243 kubenswrapper[2575]: E0728 21:06:04.229135    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6\": container with ID starting with 6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6 not found: ID does not exist" containerID="6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6"
Jul 28 21:06:04.229241 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.229165    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6"} err="failed to get container status \"6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6\": rpc error: code = NotFound desc = could not find container \"6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6\": container with ID starting with 6dc2db562cb3ae26c75720d87995e9d1370da58bbb5b039d05bd97e7380540d6 not found: ID does not exist"
Jul 28 21:06:04.235298 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.235276    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"]
Jul 28 21:06:04.238055 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.238037    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-35c841-654d849f6d-qfl8j"]
Jul 28 21:06:04.256141 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.256118    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"]
Jul 28 21:06:04.258605 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.258583    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-35c841-5f9c7c84b6-d2l7t"]
Jul 28 21:06:04.838601 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.838521    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08c14450-d207-40d6-8e7c-4bb0140145c7" path="/var/lib/kubelet/pods/08c14450-d207-40d6-8e7c-4bb0140145c7/volumes"
Jul 28 21:06:04.838979 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:04.838873    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b479785-71d5-4bd2-9445-4780b792ca49" path="/var/lib/kubelet/pods/8b479785-71d5-4bd2-9445-4780b792ca49/volumes"
Jul 28 21:06:09.865558 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:09.865521    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:06:11.803525 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:11.803491    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:06:16.356003 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:16.355977    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cqcqj_3c3fde1b-a971-4276-9a68-a304440d6094/manager/0.log"
Jul 28 21:06:16.492316 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:16.492191    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-h677p_37e161ce-b65e-42a3-ab4a-a669bb4a3880/manager/0.log"
Jul 28 21:06:16.621877 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:16.621812    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5dc5fc9c88-n2wmz_9237cb59-f382-4351-b6f6-7b6b1c5cf816/manager/0.log"
Jul 28 21:06:16.748485 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:16.748454    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lb5h7_59ee841f-e97b-4f95-b29b-e6a7fc27965d/server/0.log"
Jul 28 21:06:16.869243 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:16.869201    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-g7mnm_b3f12b87-4fc9-4e03-b35f-cc1daac7e73e/manager/0.log"
Jul 28 21:06:17.244241 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:17.244210    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv_c7a4a1da-4f55-4800-b382-2291328d3740/copy-manifests/0.log"
Jul 28 21:06:17.258135 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:17.258109    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv_c7a4a1da-4f55-4800-b382-2291328d3740/manager/0.log"
Jul 28 21:06:17.622788 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:17.622714    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8b5db48f9-l56bc_45ba6fba-6845-4fd2-8248-b0b5bfd51fe2/maas-api/0.log"
Jul 28 21:06:18.072486 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:18.072459    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-pm6q4_1464ff9a-b5f8-4849-a493-2b58539d7284/postgres/0.log"
Jul 28 21:06:19.519811 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:19.519781    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-p7d7b_44fd7dfb-b62c-417d-9505-29daa9d843a3/manager/0.log"
Jul 28 21:06:19.641933 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:19.641903    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-jz6wv_44b7304b-3f3d-479b-a6f6-1923e2070a0b/manager/0.log"
Jul 28 21:06:19.877135 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:19.877065    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2pwvq_a0f6de5b-f022-4d5e-af24-25c3b0fca65d/registry-server/0.log"
Jul 28 21:06:19.998802 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:19.998772    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qmqg5_382a59c7-d2c2-475f-8dde-ded1e0004843/manager/0.log"
Jul 28 21:06:20.237992 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:20.237960    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6t2qp_dda2a715-536e-4f18-bc21-402087b7960a/manager/0.log"
Jul 28 21:06:20.844975 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:20.844946    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bb564c89-gwmd4_59a11572-84d3-4acd-9824-243c2757d16f/kube-auth-proxy/0.log"
Jul 28 21:06:21.208359 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:21.208330    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9dgtc_a8f553c8-585e-4ab2-91b0-926fc1cf097c/payload-pre-processing/0.log"
Jul 28 21:06:21.918670 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:21.918644    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2bt48_63d1ad1d-a6d9-45bc-b189-40bd972dd297/main/0.log"
Jul 28 21:06:22.151055 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:22.151025    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj2grm_822fe93d-94ec-4d7b-96c5-de0735b61c81/main/0.log"
Jul 28 21:06:22.270030 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:22.269936    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-v2n2g_aaa44c39-d75e-40ac-ad27-262b03a2550c/main/0.log"
Jul 28 21:06:27.595196 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595158    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cm86f/must-gather-ftm67"]
Jul 28 21:06:27.595591 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595531    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b479785-71d5-4bd2-9445-4780b792ca49" containerName="payload-pre-processing"
Jul 28 21:06:27.595591 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595542    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b479785-71d5-4bd2-9445-4780b792ca49" containerName="payload-pre-processing"
Jul 28 21:06:27.595591 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595562    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af2bce9-d428-49fb-9b4e-c847785c9541" containerName="payload-pre-processing"
Jul 28 21:06:27.595591 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595568    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af2bce9-d428-49fb-9b4e-c847785c9541" containerName="payload-pre-processing"
Jul 28 21:06:27.595591 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595577    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c1d6a7f-2f16-4cbe-add6-375b70062942" containerName="maas-api"
Jul 28 21:06:27.595591 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595583    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c1d6a7f-2f16-4cbe-add6-375b70062942" containerName="maas-api"
Jul 28 21:06:27.595591 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595589    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="963d9f24-4506-4aa7-8981-a83213193d83" containerName="maas-api"
Jul 28 21:06:27.595591 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595595    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="963d9f24-4506-4aa7-8981-a83213193d83" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595602    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49b11ba-4f16-4a32-b515-74c95848d2a7" containerName="payload-processing"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595607    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49b11ba-4f16-4a32-b515-74c95848d2a7" containerName="payload-processing"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595616    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9a8d93-4f56-46bd-ac4f-eef5fdd90289" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595621    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9a8d93-4f56-46bd-ac4f-eef5fdd90289" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595627    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa05c72e-29ca-41bf-bb0f-f5c5106ba615" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595632    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa05c72e-29ca-41bf-bb0f-f5c5106ba615" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595641    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbfd028-8b80-472d-a326-2a5cbe95ee57" containerName="payload-pre-processing"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595646    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbfd028-8b80-472d-a326-2a5cbe95ee57" containerName="payload-pre-processing"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595652    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c14450-d207-40d6-8e7c-4bb0140145c7" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595657    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c14450-d207-40d6-8e7c-4bb0140145c7" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595707    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c14450-d207-40d6-8e7c-4bb0140145c7" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595714    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9a8d93-4f56-46bd-ac4f-eef5fdd90289" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595720    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c1d6a7f-2f16-4cbe-add6-375b70062942" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595726    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddbfd028-8b80-472d-a326-2a5cbe95ee57" containerName="payload-pre-processing"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595731    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b479785-71d5-4bd2-9445-4780b792ca49" containerName="payload-pre-processing"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595737    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af2bce9-d428-49fb-9b4e-c847785c9541" containerName="payload-pre-processing"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595742    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa05c72e-29ca-41bf-bb0f-f5c5106ba615" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595746    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="963d9f24-4506-4aa7-8981-a83213193d83" containerName="maas-api"
Jul 28 21:06:27.595845 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.595752    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49b11ba-4f16-4a32-b515-74c95848d2a7" containerName="payload-processing"
Jul 28 21:06:27.600326 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.600308    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cm86f/must-gather-ftm67"
Jul 28 21:06:27.603412 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.603391    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cm86f\"/\"default-dockercfg-xkx86\""
Jul 28 21:06:27.603866 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.603846    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cm86f\"/\"openshift-service-ca.crt\""
Jul 28 21:06:27.604610 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.604593    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cm86f\"/\"kube-root-ca.crt\""
Jul 28 21:06:27.615952 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.615931    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cm86f/must-gather-ftm67"]
Jul 28 21:06:27.641078 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.641043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5b2c2773-76ce-4514-9636-e3139d67c720-must-gather-output\") pod \"must-gather-ftm67\" (UID: \"5b2c2773-76ce-4514-9636-e3139d67c720\") " pod="openshift-must-gather-cm86f/must-gather-ftm67"
Jul 28 21:06:27.641214 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.641103    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrhn\" (UniqueName: \"kubernetes.io/projected/5b2c2773-76ce-4514-9636-e3139d67c720-kube-api-access-jvrhn\") pod \"must-gather-ftm67\" (UID: \"5b2c2773-76ce-4514-9636-e3139d67c720\") " pod="openshift-must-gather-cm86f/must-gather-ftm67"
Jul 28 21:06:27.662122 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.662087    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:06:27.742352 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.742322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5b2c2773-76ce-4514-9636-e3139d67c720-must-gather-output\") pod \"must-gather-ftm67\" (UID: \"5b2c2773-76ce-4514-9636-e3139d67c720\") " pod="openshift-must-gather-cm86f/must-gather-ftm67"
Jul 28 21:06:27.742513 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.742399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrhn\" (UniqueName: \"kubernetes.io/projected/5b2c2773-76ce-4514-9636-e3139d67c720-kube-api-access-jvrhn\") pod \"must-gather-ftm67\" (UID: \"5b2c2773-76ce-4514-9636-e3139d67c720\") " pod="openshift-must-gather-cm86f/must-gather-ftm67"
Jul 28 21:06:27.742664 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.742647    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5b2c2773-76ce-4514-9636-e3139d67c720-must-gather-output\") pod \"must-gather-ftm67\" (UID: \"5b2c2773-76ce-4514-9636-e3139d67c720\") " pod="openshift-must-gather-cm86f/must-gather-ftm67"
Jul 28 21:06:27.752839 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.752810    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrhn\" (UniqueName: \"kubernetes.io/projected/5b2c2773-76ce-4514-9636-e3139d67c720-kube-api-access-jvrhn\") pod \"must-gather-ftm67\" (UID: \"5b2c2773-76ce-4514-9636-e3139d67c720\") " pod="openshift-must-gather-cm86f/must-gather-ftm67"
Jul 28 21:06:27.809419 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.809387    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log" limit=1024
Jul 28 21:06:27.909430 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:27.909392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cm86f/must-gather-ftm67"
Jul 28 21:06:28.040502 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:28.040474    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cm86f/must-gather-ftm67"]
Jul 28 21:06:28.043078 ip-10-0-140-243 kubenswrapper[2575]: W0728 21:06:28.043053    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b2c2773_76ce_4514_9636_e3139d67c720.slice/crio-1b5c5b7c3ff71ca18fb9bc51f7b72cc1ab5de808e5ffc51fca204a693d33dda0 WatchSource:0}: Error finding container 1b5c5b7c3ff71ca18fb9bc51f7b72cc1ab5de808e5ffc51fca204a693d33dda0: Status 404 returned error can't find the container with id 1b5c5b7c3ff71ca18fb9bc51f7b72cc1ab5de808e5ffc51fca204a693d33dda0
Jul 28 21:06:28.044770 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:28.044753    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 21:06:28.332676 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:28.332593    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cm86f/must-gather-ftm67" event={"ID":"5b2c2773-76ce-4514-9636-e3139d67c720","Type":"ContainerStarted","Data":"1b5c5b7c3ff71ca18fb9bc51f7b72cc1ab5de808e5ffc51fca204a693d33dda0"}
Jul 28 21:06:29.341795 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:29.341689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cm86f/must-gather-ftm67" event={"ID":"5b2c2773-76ce-4514-9636-e3139d67c720","Type":"ContainerStarted","Data":"6f60c7f064f2a864bfbafb6242eb705bed36b7cb236901a2167b2fd42404cf26"}
Jul 28 21:06:29.341795 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:29.341750    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cm86f/must-gather-ftm67" event={"ID":"5b2c2773-76ce-4514-9636-e3139d67c720","Type":"ContainerStarted","Data":"7e8a09d9ff7330be1f7f3146f181a2d71a6a3dcaaf8c15f19badaefe6cb0b529"}
Jul 28 21:06:29.358623 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:29.358562    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cm86f/must-gather-ftm67" podStartSLOduration=1.480829429 podStartE2EDuration="2.358541796s" totalImagesPullingTime="877.712367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:06:29.356480108 +0000 UTC m=+3488.970717241" watchObservedRunningTime="2026-07-28 21:06:29.358541796 +0000 UTC m=+3488.972778930"
Jul 28 21:06:30.572052 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:30.572009    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j2kp2_d953ada0-b545-41e5-901e-4b69b4d33bae/global-pull-secret-syncer/0.log"
Jul 28 21:06:30.673954 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:30.673925    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d5z5k_598f2d8e-b1f5-4e33-8dcf-30400b6357c1/konnectivity-agent/0.log"
Jul 28 21:06:30.836541 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:30.836463    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-243.ec2.internal_6bbf87afc187fb5805cf1fa6a52b872b/haproxy/0.log"
Jul 28 21:06:36.117188 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:36.117099    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-p7d7b_44fd7dfb-b62c-417d-9505-29daa9d843a3/manager/0.log"
Jul 28 21:06:36.156352 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:36.156323    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-jz6wv_44b7304b-3f3d-479b-a6f6-1923e2070a0b/manager/0.log"
Jul 28 21:06:36.245172 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:36.245096    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2pwvq_a0f6de5b-f022-4d5e-af24-25c3b0fca65d/registry-server/0.log"
Jul 28 21:06:36.481270 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:36.481211    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qmqg5_382a59c7-d2c2-475f-8dde-ded1e0004843/manager/0.log"
Jul 28 21:06:36.597756 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:36.597565    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6t2qp_dda2a715-536e-4f18-bc21-402087b7960a/manager/0.log"
Jul 28 21:06:38.506770 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:38.506742    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bq8qv_92d3c5f9-8862-413f-8924-0ddd15bb4558/node-exporter/0.log"
Jul 28 21:06:38.527680 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:38.527618    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bq8qv_92d3c5f9-8862-413f-8924-0ddd15bb4558/kube-rbac-proxy/0.log"
Jul 28 21:06:38.556934 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:38.556906    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bq8qv_92d3c5f9-8862-413f-8924-0ddd15bb4558/init-textfile/0.log"
Jul 28 21:06:40.242188 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.242150    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"]
Jul 28 21:06:40.247801 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.247773    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.255373 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.255347    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"]
Jul 28 21:06:40.366843 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.366785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-sys\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.366843 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.366846    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-lib-modules\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.367109 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.366879    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-podres\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.367109 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.366926    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-proc\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.367109 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.366972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gj9\" (UniqueName: \"kubernetes.io/projected/5988be32-883d-42f3-8284-b5835389afee-kube-api-access-w9gj9\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.468253 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.468193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-proc\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.468447 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.468315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gj9\" (UniqueName: \"kubernetes.io/projected/5988be32-883d-42f3-8284-b5835389afee-kube-api-access-w9gj9\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.468447 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.468392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-sys\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.468447 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.468427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-lib-modules\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.468633 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.468466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-podres\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.468633 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.468578    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-sys\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.468744 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.468637    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-proc\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.468801 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.468741    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-lib-modules\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.468801 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.468768    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5988be32-883d-42f3-8284-b5835389afee-podres\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.476977 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.476925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gj9\" (UniqueName: \"kubernetes.io/projected/5988be32-883d-42f3-8284-b5835389afee-kube-api-access-w9gj9\") pod \"perf-node-gather-daemonset-6p5pt\" (UID: \"5988be32-883d-42f3-8284-b5835389afee\") " pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.562684 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.562598    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:40.791403 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:40.791221    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"]
Jul 28 21:06:41.434770 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:41.433755    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt" event={"ID":"5988be32-883d-42f3-8284-b5835389afee","Type":"ContainerStarted","Data":"7767a2be792953eeb3a5c63b91d545ecd02c9909f20c214d573fa17f6e7da877"}
Jul 28 21:06:41.434770 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:41.433823    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt" event={"ID":"5988be32-883d-42f3-8284-b5835389afee","Type":"ContainerStarted","Data":"f6a2c3322f2c05aaf4a48c7a8af02611805c42c21addd6d88be0f6c74efe10ab"}
Jul 28 21:06:41.434770 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:41.434632    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:41.449944 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:41.449882    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt" podStartSLOduration=1.4498655 podStartE2EDuration="1.4498655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 21:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:06:41.44954053 +0000 UTC m=+3501.063777663" watchObservedRunningTime="2026-07-28 21:06:41.4498655 +0000 UTC m=+3501.064102634"
Jul 28 21:06:43.151749 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:43.151654    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ntjln_48abf100-5828-4ae4-bb41-d90680574b26/dns/0.log"
Jul 28 21:06:43.172249 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:43.172203    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ntjln_48abf100-5828-4ae4-bb41-d90680574b26/kube-rbac-proxy/0.log"
Jul 28 21:06:43.332894 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:43.332861    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n87ww_f52174fd-3b1a-465b-b0f4-4068deab6013/dns-node-resolver/0.log"
Jul 28 21:06:43.880042 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:43.880005    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-66674cb6c9-wngfz_0864b1c1-412c-4667-9cbd-58ae7d7f0de3/registry/0.log"
Jul 28 21:06:43.902323 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:43.902291    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4db9q_88a7f2ff-c2cc-4b29-8bf2-66861c7931bc/node-ca/0.log"
Jul 28 21:06:45.124019 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:45.123992    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bb564c89-gwmd4_59a11572-84d3-4acd-9824-243c2757d16f/kube-auth-proxy/0.log"
Jul 28 21:06:45.260822 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:45.260787    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9dgtc_a8f553c8-585e-4ab2-91b0-926fc1cf097c/payload-pre-processing/0.log"
Jul 28 21:06:46.168391 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:46.168362    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-spn79_103df0b3-09df-4d38-8d49-052a14d5556a/serve-healthcheck-canary/0.log"
Jul 28 21:06:46.759173 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:46.759143    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rgnhx_898ae9a8-22e1-41de-90c4-ad47d2ac4e47/kube-rbac-proxy/0.log"
Jul 28 21:06:46.778520 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:46.778492    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rgnhx_898ae9a8-22e1-41de-90c4-ad47d2ac4e47/exporter/0.log"
Jul 28 21:06:46.799360 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:46.799328    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rgnhx_898ae9a8-22e1-41de-90c4-ad47d2ac4e47/extractor/0.log"
Jul 28 21:06:48.457642 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:48.457610    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cm86f/perf-node-gather-daemonset-6p5pt"
Jul 28 21:06:48.848477 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:48.848400    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cqcqj_3c3fde1b-a971-4276-9a68-a304440d6094/manager/0.log"
Jul 28 21:06:49.119130 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:49.119033    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-h677p_37e161ce-b65e-42a3-ab4a-a669bb4a3880/manager/0.log"
Jul 28 21:06:49.291844 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:49.291803    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5dc5fc9c88-n2wmz_9237cb59-f382-4351-b6f6-7b6b1c5cf816/manager/0.log"
Jul 28 21:06:49.309708 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:49.309682    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lb5h7_59ee841f-e97b-4f95-b29b-e6a7fc27965d/server/0.log"
Jul 28 21:06:49.398165 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:49.398136    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-g7mnm_b3f12b87-4fc9-4e03-b35f-cc1daac7e73e/manager/0.log"
Jul 28 21:06:49.603157 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:49.603119    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv_c7a4a1da-4f55-4800-b382-2291328d3740/manager/0.log"
Jul 28 21:06:49.623572 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:49.623547    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74ff7fbd5d-lw6vv_c7a4a1da-4f55-4800-b382-2291328d3740/copy-manifests/0.log"
Jul 28 21:06:51.065905 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:51.065875    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bb564c89-gwmd4_59a11572-84d3-4acd-9824-243c2757d16f/kube-auth-proxy/0.log"
Jul 28 21:06:51.196473 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:51.196372    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9dgtc_a8f553c8-585e-4ab2-91b0-926fc1cf097c/payload-pre-processing/0.log"
Jul 28 21:06:52.893531 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:52.893502    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bb564c89-gwmd4_59a11572-84d3-4acd-9824-243c2757d16f/kube-auth-proxy/0.log"
Jul 28 21:06:53.008477 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:53.008404    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9dgtc_a8f553c8-585e-4ab2-91b0-926fc1cf097c/payload-pre-processing/0.log"
Jul 28 21:06:56.050188 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:56.050158    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-6db8p_4d05786a-a870-4963-931a-9bfd0abebb1c/migrator/0.log"
Jul 28 21:06:56.070072 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:56.070043    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-6db8p_4d05786a-a870-4963-931a-9bfd0abebb1c/graceful-termination/0.log"
Jul 28 21:06:57.377874 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:57.377842    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5b5lv_376809ed-3882-4375-9c40-2a004901780c/kube-multus/0.log"
Jul 28 21:06:57.542346 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:57.542318    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tjdxd_ac142701-e24f-4e3f-90d4-0367bbafa10c/kube-multus-additional-cni-plugins/0.log"
Jul 28 21:06:57.562737 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:57.562714    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tjdxd_ac142701-e24f-4e3f-90d4-0367bbafa10c/egress-router-binary-copy/0.log"
Jul 28 21:06:57.583293 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:57.583267    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tjdxd_ac142701-e24f-4e3f-90d4-0367bbafa10c/cni-plugins/0.log"
Jul 28 21:06:57.602431 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:57.602407    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tjdxd_ac142701-e24f-4e3f-90d4-0367bbafa10c/bond-cni-plugin/0.log"
Jul 28 21:06:57.624813 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:57.624790    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tjdxd_ac142701-e24f-4e3f-90d4-0367bbafa10c/routeoverride-cni/0.log"
Jul 28 21:06:57.644940 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:57.644914    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tjdxd_ac142701-e24f-4e3f-90d4-0367bbafa10c/whereabouts-cni-bincopy/0.log"
Jul 28 21:06:57.665297 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:57.665273    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tjdxd_ac142701-e24f-4e3f-90d4-0367bbafa10c/whereabouts-cni/0.log"
Jul 28 21:06:57.907719 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:57.907654    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6kmfb_3ed50e77-948a-4cbd-ae6e-920b8c190c29/network-metrics-daemon/0.log"
Jul 28 21:06:57.926087 ip-10-0-140-243 kubenswrapper[2575]: I0728 21:06:57.926060    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6kmfb_3ed50e77-948a-4cbd-ae6e-920b8c190c29/kube-rbac-proxy/0.log"